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The  Economic  Evaluation  of  Alternatives  (EEoA): 
Rethinking  the  Application  of  Cost-effectiveness  Analysis, 
Multi-criteria  Decision-making  (MCDM)  and  the  Analysis  of 
Alternatives  (AoA)  in  Defense  Procurement 


Presenter:  Presenter:  Dr.  Francois  Melese  joined  the  Naval  Postgraduate  School  in  1987  and  today  is 
Professor  of  Economics  at  the  Defense  Resources  Management  Institute  (DRMI).  He  has  consulted 
extensively,  most  recently  with  the  Joint  Staff  and  the  OSD.  In  2008,  he  helped  edit  the  DoD’s  first 
Strategic  Management  Plan.  He  has  published  extensively  on  a  variety  of  topics,  including  a  co-authored 
paper  entitled  “A  New  Management  Model  for  Government.”  Results  in  implementing  that  model  for  the 
Joint  Staff  were  recently  published  in  the  Armed  Forces  Comptroller.  At  the  request  of  NATO  HQ,  Melese 
has  represented  the  US  as  an  expert  in  defense  management  and  public  budgeting  throughout  Europe. 
He  recently  organized  a  major  NATO  meeting  in  Monterey  on  “Building  Integrity  and  Defense  Institution 
Building.” 


Abstract 

Our  primary  goal  is  to  improve  public  investment  decisions  by  providing  defense 
analysts  and  acquisition  officials  a  comprehensive  set  of  approaches  to  structure  an  “Economic 
Evaluation  of  Alternatives”  (EEoA).  This  study  identifies  a  significant  weakness  in  the  Multi¬ 
criteria  Decision-making  (MCDM)  approach  that  currently  underpins  many  contemporary  AoAs. 
While  MCDM  techniques,  and  therefore  most  AoAs,  correctly  focus  on  lifecycle  costs  and 
operational  effectiveness  of  alternatives,  “Affordability”  is  often  only  implicitly  addressed  in  the 
final  stages  of  the  analysis.  In  contrast,  the  adoption  of  EEoA  encourages  decision-makers  to 
include  affordability  explicitly  and  up-front  in  the  AoA.  This  requires  working  with  vendors  to 
build  alternatives  based  on  different  funding  (budget/affordability)  scenarios.  The  key  difference 
between  the  traditional  MCDM  approach  to  AoAs  and  the  EEoA  approach  is  that  instead  of 
modeling  alternatives  from  competing  vendors  as  points  in  cost-effectiveness  space,  EEoA 
models  alternatives  as  functions  of  optimistic,  pessimistic,  and  most  likely  funding  (budget) 
scenarios.  The  Decision  Map  offered  to  practitioners  to  structure  EEoAs  provides  a  unique 
opportunity  to  achieve  a  significant  defense  acquisition  reform — to  coordinate  the  requirements 
generation  system  (JCIDS),  Defense  Acquisition  System  (DAS),  and  PPBE  process,  to  lower 
the  costs  of  defense  investments,  and  improve  performance  and  schedules. 

Introduction  to  the  Problem:  Making  the  Case  for 
“Affordability” 

Our  nation’s  security,  billions  of  taxpayer  dollars,  and  the  survival  of  our  soldiers  can  all 
hinge  on  an  Analysis  of  Alternatives  (AoA).1  Routinely  conducted  by  the  US  Department  of 
Defense  (DoD),  the  AoA  is  a  key  component  of  the  defense  acquisition  process.  Investment 


1  This  study  often  uses  the  term  “Analysis  of  Alternatives”  (AoA)  in  its  broad,  generic  sense.  Although 
focused  on  defense  acquisition,  the  results  of  the  study  apply  to  any  public-sector  procurement.  It  should 
be  clear  in  context  when  the  term  AoAs  references  major  defense  acquisition  programs  (MDAPs)  as 
opposed  to  the  acquisition  of  major  automated  information  systems  (MAISs). 
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decisions  supported  by  AoAs  help  shape  future  forces,  influence  defense  spending,  and 
occasionally  transform  the  defense  industry. 

This  study  points  to  a  significant  weakness  in  the  Multiple-criteria  Decision-making 
(MCDM)  approach  that  underpins  many  contemporary  AoAs.  The  weakness  is  that  while  MCDM 
techniques,  and  therefore  most  AoAs,  correctly  focus  on  lifecycle  costs  and  the  operational 
effectiveness  of  individual  alternatives,  “Affordability”  is  an  after-thought,  often  only  implicitly 
addressed  through  a  weight  assigned  to  costs. 

In  contrast,  the  approach  recommended  in  this  study  encourages  analysts  and  decision¬ 
makers  to  include  affordability  explicitly  in  the  AoA.  This  requires  working  with  vendors  to  build 
alternatives  based  on  different  funding  (budget/affordability)  scenarios.  Supported  by  a  static, 
deterministic,  multi-stage,  constrained,  optimization  micro-economic  production  (procurement 
auction)  model  described  in  Section  3  (with  the  math  relegated  to  the  Mathematical  Appendix 
available  upon  request),  this  “Economic  Evaluation  of  Alternatives”  (EEoA)  explicitly  addresses 
affordability  up-front.  The  key  difference  between  the  MCDM  approach  to  AoAs  and  the  EEoA 
approach  is  that,  instead  of  modeling  decision  alternatives  from  competing  vendors  as  points  in 
cost-effectiveness  space,  EEoA  models  alternatives  as  functions  of  optimistic,  pessimistic,  and 
most  likely  funding  (resource/budget)  scenarios.  Given  the  current  financial  crisis  and  future 
public-spending  challenges,  affordability  is  a  growing  concern.  As  a  consequence,  it  is 
imperative  that  the  DoD  gets  the  best  value  for  every  dollar  it  invests  in  major  defense 
acquisition  programs  (MDAPs)  or  major  automated  information  systems  (MAISs). 

A  brief  review  of  the  DoD’s  high-level,  fiscally  constrained  budget  development  and 
acquisition  systems  highlights  the  key  role  that  affordability  needs  to  play  up-front  in  any  AoA. 
The  Planning,  Programming,  Budgeting  and  Execution  (PPBE)  process  is  the  principal  decision 
support  system  used  by  the  DoD  to  provide  the  best  possible  mix  of  forces,  equipment,  and 
support  within  fiscal  constraints.  Two  other  major  decision  support  systems  complement  the 
PPBE  process:  a  requirements  generation  system  called  the  Joint  Capabilities  Integration  and 
Development  System  (JCIDS)  and  the  Defense  Acquisition  System  (DAS). 

Based  on  strategic-level  guidance  (the  National  Security  Strategy,  National  Military 
Strategy,  Quadrennial  Defense  Review,  Strategic  Planning  Guidance,  etc.),  the  requirements 
generation  system  reviews  existing  and  proposed  capabilities  and  identifies  critical  capability 
gaps.  To  fill  those  capability  gaps,  senior  leadership  examines  the  full  range  of  “doctrine, 
organization,  training,  materiel,  leadership  and  education,  personnel  and  facilities”  (DOTMLPF) 
(CJCS,  2007,  p.  A-1 ;  USD  (AT&L),  2008,  p.  14). 

Whenever  a  “materiel”  solution  is  recommended,  prospective  military  investments  are 
identified  that  serve  as  the  basis  for  AoAs  that  underpin  the  development  of  new  acquisition 
programs  in  the  Defense  Acquisition  System  (DAS).  The  DAS  provides  principles  and  policies 
that  govern  major  defense  acquisition  decisions  and  milestones.  To  ensure  transparency  and 
accountability,  and  to  promote  efficiency  and  effectiveness,  various  instructions  (e.g.,  FAR, 
DFARS,  DoD  Directive  5000.01,  DoD  Instruction  5000.02,  etc.)  specify  statutory  and  regulatory 
reports  (e.g.,  AoAs)  and  other  information  requirements  for  each  milestone  and  decision  point. 

The  primary  purpose  of  PPBE  is  to  make  hard  choices  among  alternative  military 
investments  necessary  for  national  security  within  fiscal  constraints.  As  we  identify  alternative 
materiel  investments  that  can  fill  current  capability  gaps,  the  requirements  generation  process 
(JCIDS)  naturally  fits  into  the  Planning  phase  of  PPBE. 
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The  first  step  in  any  investment  analysis  is  to  identify  the  derived  demand  for  a  key 
capability,  program,  or  project.  This  is  accomplished  through  the  DoD’s  requirements  generation 
system  (JCIDS).  Ideally,  user  demands  are  expressed  and  refined  in  the  Planning  phase  of  the 
PPBE  process.  MDAP  and  MAIS  proposals  that  emerge  from  JCIDS  and  the  Planning  process 
enter  the  Defense  Acquisition  System  (DAS)  and  are  incorporated  in  the  Programming  phase  of 
PPBE. 


The  Planning  phase  of  PPBE  establishes  fiscally  constrained  guidance  and  priorities  for 
military  forces,  including  readiness,  sustainability  and  modernization.  This  guidance  provides 
direction  for  DoD  Components  (military  departments  and  defense  agencies)  to  develop  their 
individual  program  proposals  or  their  Program  Objective  Memorandum  (POM)  in  the 
Programming  phase.  The  POM  details  resource-allocation  decisions  (funding,  personnel,  etc.) 
proposed  by  each  Component  for  its  programs,  projected  six  years  into  the  future.  DAS  data  for 
major  defense  acquisitions  generally  includes  lifecycle  cost  estimates  that  project  well  beyond 
the  six  years  of  the  POM. 

Senior  leadership  in  the  OSD  and  the  Joint  Staff  subsequently  review  each  Component 
POM  to  ensure  it  satisfies  the  Planning  guidance,  and  that  it  can  be  integrated  into  effective  and 
affordable  overall  defense  programs.  The  Budgeting  phase  of  PPBE  occurs  concurrently  with 
the  Programming  phase. 

The  Budgeting  phase  converts  the  Programming  phase’s  (output-oriented)  view  into  the 
(input-oriented)  format  required  by  Congressional  appropriation  structures.  While  the  DoD’s 
biennial  defense  budget  projects  funding  only  two  years  into  the  future,  it  includes  more  financial 
detail  than  the  POMs.  The  Under  Secretary  of  Defense  Comptroller  and  the  Office  of 
Management  and  Budget  (OMB)  are  responsible  for  reviewing  budget  submissions  to  ensure 
programs  are  affordable,  i.e.,  satisfy  current  fiscal  constraints.  The  GAO  recognizes  the  major 
challenges  faced  by  the  DoD  to  “achieve  a  balanced  mix  of  weapon  systems  that  are  affordable” 
(GAO,  2009,  p.  5). 2 

The  primary  focus  of  Multi-criteria  Decision-making  (MCDM),  as  traditionally  applied  in 
AoAs,  is  to  evaluate  the  lifecycle  costs  and  operational  effectiveness  of  alternative  defense 
investments.  “An  AoA  is  an  analytical  comparison  of  the  operational  effectiveness,  suitability, 
and  Life-Cycle  Cost  of  alternatives  that  satisfy  established  Capability  needs”  (DoD,  2006,  July  7, 
Section  3.3).  This  study  emphasizes  another  key  aspect — “Affordability.” 

In  helping  generate  investment  alternatives,  and  illuminating  advantages  and 
disadvantages  of  those  alternatives,  AoAs  have  the  potential  to  contribute  to  requirements 
generation  in  the  Planning  phase  of  PPBE,  and  through  DAS  decision  milestones,  also  in  the 
Programming  phase  of  PPBE.  However,  according  to  the  GAO:  “while  JCIDS  provides  a 
framework  for  reviewing  and  validating  needs  [. . .]  the  vast  majority  of  capability  proposals  that 
enter  the  JCIDS  process  are  validated  or  approved  without  accounting  for  resources  [funding, 
budgets]  [...]  that  will  be  needed  to  acquire  the  desired  capabilities”  (GAO,  2009,  p.  6). 3 


2  According  to  the  Government  Accountability  Office  (GAO),  over  the  next  5  years,  the  DoD  plans  to 
spend  more  than  $357  billion  on  development  and  procurement  of  major  defense  acquisition  programs 
(GAO,  2009,  p.  4). 

3  “A  2008  DoD  directive  established  nine  joint  capability-area  portfolios,  each  managed  by  civilian  and 
military  co-leads  [...].  However,  without  [...]  control  over  resources  [funding/budgets],  the  department  is  at 


DEFENSE  ACQUISITION  IN  TRANSITION 


We  believe  the  GAO’s  results  reflect  a  weakness  in  the  way  AoAs  have  traditionally 
been  structured.  While  AoAs  provide  a  sharp  focus  on  cost  and  effectiveness  estimates  of 
competing  alternatives,  the  affordability  (funding/budget  realities)  of  the  overall  program  is  at 
best  implicit,  and  usually  conducted  ex-post.4  This  is  reflected  in  the  GAO’s  concern  that  “at  the 
program  level,  the  key  cause  of  poor  outcomes  is  the  approval  of  programs  with  business  cases 
[e.g.,  AoAs]  that  contain  inadequate  knowledge  about  [...]  resources  [funding]  [...]  needed  to 
execute  them”  (2009,  p.  7). 

Yet  DoD  Directive  5000.01  explicitly  states  that,  “All  participants  in  the  acquisition 
system  shall  recognize  the  reality  of  fiscal  constraints  [. . .].  DoD  components  shall  plan  [. . .] 
based  on  realistic  projections  of  the  dollars  [. . .]  likely  to  be  available  [. .  .and]  the  user  shall 
address  affordability  in  establishing  capability  needs”  (USD  (AT&L),  2007,  Enclosure  1,  p.  5). 

For  all  major  (ACAT  1)  defense  acquisition  programs,  an  AoA  is  required  at  key 
milestone  decision  points  (i.e. ,  A,  B,  C).  Affordability  assessments  are  required  at  Milestones  B 
and  C  for  major  defense  acquisition  programs  and  automated  information  systems  (USD 
(AT&L),  2008,  Enclosure  4,  p.  40). 

According  to  the  Defense  Acquisition  Guidebook,  the  purpose  of  an  affordability 
assessment  is  to  demonstrate  that  the  program’s  projected  funding  requirements  are  realistic 
and  achievable.5  “In  general,  the  assessment  should  address  program  funding  over  the  six-year 
programming  period,  and  several  years  beyond.  The  assessment  should  also  show  how  the 
projected  funding  fits  within  the  overall  DoD  Component  plan”  (DoD,  2006,  July  7,  Section 
3.2.2.). 6 

The  Economic  Evaluation  of  Alternatives  (EEoA)  essentially  embeds  an  explicit 
affordability  assessment  into  an  AoA.  In  preparing  affordability  assessments,  one  possible 
source  of  data  is  the  Future  Years  Defense  Program  (FYDP).7  According  to  the  Defense 
Acquisition  Guidebook,  affordability  assessments  should  provide  details  as  to  how  excess 


risk  [...]  of  not  knowing  if  its  systems  are  being  developed  within  available  resources  [funding/budgets]” 
(GAO,  2009,  p.  11). 

4  “Typically,  the  last  analytical  section  of  the  AoA  plan  deals  with  the  planned  approach  for  the  cost- 
effectiveness  comparisons  of  the  study  alternatives”  (DoD,  2006,  July  7,  Section  3.3).  Note  that  there  is 
no  mention  of  “affordability,”  but  instead  only  an  ex-post  cost-effectiveness  trade-off  that  implies  a 
concern  for  affordability.  Moreover,  this  trade-off  occurs  at  the  end  of  a  process  in  which  alternatives 
under  consideration  have  been  developed  independently  of  any  cost/budget/funding/affordability 
constraint.  The  US  Marine  Corps  (PA&E)  has  a  similar  approach  to  structuring  an  AoA. 

5  Since  this  assessment  requires  a  DoD  Component  corporate  perspective,  the  affordability  assessment 
should  not  be  prepared  by  the  program  manager  nor  should  it  rely  too  heavily  on  the  user.  It  requires  a 
higher-level  perspective  capable  of  balancing  budget  trade-offs  (affordability)  across  a  set  of  users  (2006, 
July  7,  Section  3.2.2). 

6  A  first  step  in  the  program’s  affordability  assessment  is  to  portray  the  projected  annual  modernization 
funding  (RDT&E  plus  procurement,  measured  as  TOA)  in  constant  dollars  for  the  six-year  programming 
period  and  for  twelve  years  beyond.  Similar  funding  streams  for  other  acquisition  programs  in  the  same 
mission  area  also  would  be  included.  What  remains  to  be  determined  is  whether  this  projected  funding 
growth  is  realistically  affordable  relative  to  the  DoD  Component’s  most  likely  overall  funding.  The  model  in 
this  study  proposes  structuring  the  Economic  Evaluation  of  Alternatives  not  only  for  a  most  likely  budget 
but  also  for  an  optimistic  and  pessimistic  budget. 

7  An  output  of  the  DoD’s  PPBE  process,  the  FYDP  is  an  OSD  database  that  contains  future  budget 
projections. 
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funding  demands  will  be  accommodated  by  reductions  in  other  mission  areas,  or  in  other 
accounts.8  This  Opportunity  Cost  Approach  is  the  last  of  six  ways  proposed  in  this  study  to 
structure  an  Economic  Evaluation  of  Alternatives  (EEoA). 

Nesting  the  Requirements  Generation  and  Defense  Acquisition  Systems  within  PPBE 
suggests  formulating  the  military’s  acquisition  problem  in  terms  of  identifying  and  funding 
specific  defense  investments  that  maximize  value  (performance  or  effectiveness)  for  a  given 
budget.  If  AoAs  were  framed  as  a  constrained  optimization — i.e.,  maximizing  performance 
subject  to  a  budget  constraint  (or  alternatively  minimizing  costs  subject  to  a  given  level  of 
performance) — they  could  be  used  to  support  resource-allocation  decisions  in  the  Programming 
phase  of  PPBE.9  These  dual  constrained-optimization  approaches  represent  the  first  two  of  six 
ways  proposed  in  this  study  to  structure  an  Economic  Evaluation  of  Alternatives  (EEoA). 

Unfortunately,  MCDM  techniques  typically  applied  to  structure  an  AoA  do  not  easily  lend 
themselves  to  this  interpretation.  As  a  consequence,  instead  of  being  constrained  by  budgets, 
budgets  are  more  often  the  output  of  an  AoA,  generating  and/or  supporting  so-called  “funding 
requirements.”  Our  third  approach  to  structuring  an  Economic  Evaluation  of  Alternatives  (EEoA) 
turns  this  on  its  head. 

Instead  of  generating  a  budget  through  the  AoA  process,  we  propose  that  decision¬ 
makers  or  analysts  forecast  an  optimistic,  pessimistic,  and  most  likely  budget  as  part  of  the 
PPBE  process,  and  then  approach  vendors  to  generate  alternatives  that  fit  within  that  budget 
envelope.10  This  offers  an  alternate  approach  to  defense  investment  decisions  based  on  explicit 
funding  (resource/budget/affordability)  scenarios.  This  also  supports  the  “long-standing  DoD 
policy  to  seek  full  funding  of  acquisition  programs,  based  on  the  most  likely  cost”  (DoD,  2006, 
July  7,  Chapter  3.23). 

The  primary  goal  of  this  study  is  to  improve  defense  decisions  by  bringing  the  taxpayer 
up-front  alongside  the  warfighter  in  the  defense  acquisition  process.  This  is  accomplished  by 
explicitly  introducing  an  affordability  assessment  in  EEoA  through  optimistic,  pessimistic,  and 
most  likely  funding  scenarios.  Unlike  traditional  MCDM  approaches  to  AoAs  that  focus  on  cost 
and  operational  effectiveness,  an  EEoA  adds  a  third  dimension.  It  makes  a  clear  distinction 


8  Note  that  in  the  “off-year”  of  the  biennial  PPBE  process,  DoD  Components  are  restricted  to  the  second 
year  of  the  biennial  budget  and  are  required  to  submit  Program  Change  Proposals  (PCPs)  and/or  Budget 
Change  Proposals  (BCPs)  to  account  for  any  program-cost  increases,  schedule  delays,  etc.  PCPs 
address  issues  over  a  multi-year  period,  whereas  BCPs  address  issues  focused  on  the  upcoming  budget 
year.  Moreover,  to  stay  within  fiscal  constraints,  BCPs  and  PCPs  must  identify  resource  reductions  in 
other  programs  to  offset  any  cost  growth.  This  is  similar  in  spirit  to  the  “opportunity  cost”  approach  that  we 
propose  as  one  of  six  ways  to  structure  an  EEoA. 

9  Translating  the  budget  implications  of  these  decisions  into  the  usual  Congressional  appropriation 
categories  (Military  Personnel,  Procurement,  Operations  &  Maintenance  (O&M),  Military  Construction, 
etc.)  generates  the  defense  budget  and  Future  Year  Defense  Program  (FYDP). 

10  This  is  in  the  spirit  of  the  Department  of  the  Army’s  Acquisition  Procedures,  which  explicitly  state  that 
“Cost  as  an  Independent  Variable  (CAIV)  applies  to  all  defense  acquisition  programs  [....and]  treats  cost 
as  an  input  to,  rather  than  an  output  of,  the  materiel  requirements  and  acquisition  processes.”  The  Army 
guidance  emphasizes  “CAIV  is  focused  on  [...]  meeting  operational  requirements  with  a  solution  that  is 
affordable  [...and  that  does]  not  exceed  cost  constraints  [and  to]  establish  CAIV-based  cost  objectives 
(development,  procurement,  and  sustainment  costs)  early  in  the  acquisition  process.”  Moreover,  the 
“RFP  must  [...]  solicit  from  potential  suppliers  an  approach  [...]  for  meeting  CAIV  objectives”  (DoA,  1999, 
July  15,  p.  63). 
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between  the  “lifecycle  cost”  or  “price”  of  an  alternative,  its  operational  effectiveness,  and 
“funding”  (budget  or  resources)  available. 

The  EEoA  approach  responds  to  two  fundamental  challenges  highlighted  by  the  GAO 
that  continue  to  face  the  DoD’s  Defense  Acquisition  System:  “(1)  [to  make]  better  decisions 
about  which  programs  should  be  pursued  or  not  pursued  given  existing  and  expected  funding; 
[and]  (2)  [to  develop]  an  analytical  approach  to  better  prioritize  capability  needs”  (GAO,  2009, 
March  18,  Highlights). 

In  stressing  affordability,  EEoA  offers  an  analytical  approach  that  begins  to  resolve  a 
major  concern  expressed  by  the  GAO: 

DoD’s  processes  for  identifying  war-fighter  needs  [JCIDS],  allocating  resources  [PPBE], 
and  developing  and  procuring  weapon  systems  [DAS...]  are  fragmented  [...so  that]  DoD 
commits  to  more  programs  than  resources  [budgets]  can  support  [...].  DoD  allows 
programs  to  begin  development  without  a  full  understanding  [of]  the  resources 
[budget/funding]  needed.11  (2009,  March  18,  Highlights) 

Whereas  funding  decisions  for  major  programs  take  place  throuqh  the  PPBE  process, 
the  GAO  finds  that: 

[T]he  process  does  not  produce  an  accurate  picture  of  the  department’s  resource  needs 
[funding/budget  requirements]  for  weapon  system  programs  [...].  Ultimately,  the  process 
produces  more  demand  for  new  weapon  system  programs  than  available  resources  can 
support.12  (2009,  March  18,  p.  6) 

The  EEoA  approach  proposed  in  this  study  represents  an  important  step  in  integrating 
the  DoD’s  requirements  generation  and  Defense  Acquisition  Systems  with  PPBE.  For  instance, 
in  considering  alternative  budget  scenarios  that  rely  on  the  FYDP,  it  injects  an  explicit 
constrained-optimization  approach  into  the  Defense  Acquisition  System  (DAS)  that  parallels  the 
one  already  embedded  in  the  PPBE  process.13 

In  generating  alternatives  under  optimistic,  pessimistic  and  most  likely  budget  scenarios, 
the  EEoA  requires  explicit  interaction  with  the  PPBE  process.  In  sharp  contrast  with  the  MCDM 
approach  that  underlies  most  AoAs,  the  EEoA  approaches  explicitly  identify  and  emphasize 
budgets,  funding,  and  affordability.  Ultimately,  widespread  adoption  of  the  EEoA  would 
contribute  to  the  goal  of: 

greater  consultation  between  requirements,  budget,  and  acquisition  processes  [that] 
could  help  improve  the  department’s  [...]  portfolio  of  weapon  programs  [...].  This  means 
that  decision  makers  responsible  for  weapon  system  requirements,  funding,  and 


11  “The  lack  of  early  systems  engineering,  acceptance  of  unreliable  cost  estimates  based  on  overly 
optimistic  assumptions,  failure  to  commit  full  funding,  and  the  addition  of  new  requirements  well  into  the 
acquisition  cycle  all  contribute  to  poor  outcomes”  (GAO,  2009,  March  18).  Whereas  this  study  focuses  on 
funding  risks,  Melese,  Franck,  Angelis  and  Dillard  (2007,  January)  introduce  an  economic  approach 
called  “Transaction  Cost  Analysis”  that  addresses  the  other  GAO  concerns. 

12  The  cost  of  many  programs  reviewed  by  the  GAO  exceeded  planned  funding/budget  levels  (GAO, 
2008,  July  2). 

13  Office  of  Management  and  Budget  (OMB)  Circular  A-1 1  titled  Preparation  and  Submission  of  Budget 
Estimates  is  the  official  guidance  on  the  preparation  and  submission  of  budget  estimates  to  Congress. 
The  Army’s  Acquisition  guidance  emphasizes  “the  requirement  for  presenting  the  full  funding  for  an 
acquisition  program — that  is  the  total  cost  [for]  a  given  system  as  reflected  in  the  most  recent  FYDP  [...] 
pertains  to  all  acquisition  programs”  (DoA,  1999,  July  15,  p.  41). 
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acquisition  execution  must  establish  an  investment  strategy  in  concert  [...],  assuring 
requirements  for  specific  weapon  systems  are  clearly  defined  and  achievable  given 
available  resources  [funding/budgets].  (GAO,  2008,  July  2,  p.  1 0,  14) 

The  next  section  offers  a  brief  description  and  critical  evaluation  of  the  status  quo.  We 
review  two  common  decision  criteria  used  in  cost-effectiveness  analyses.  The  first  is  the  popular 
“bang-for-the-buck”  or  Benefit/Cost  ratio.  The  second  criterion  is  essentially  a  weighted  average 
of  cost  and  effectiveness,  a  decision  rule  generated  by  the  standard  static,  deterministic  MCDM 
approach  to  cost-effectiveness  analysis  that  underpins  most  contemporary  AoAs. 

Section  3  offers  a  set  of  alternate  approaches  to  resolve  the  decision-criterion  problem. 
Six  intuitive  approaches  are  described  to  structure  an  Economic  Evaluation  of  Alternatives 
(EEoA).14  Section  4  concludes  with  a  Decision  Map  to  guide  analysts  and  decision-makers  in 
selecting  which  of  the  six  approaches  is  best  suited  for  them  to  structure  an  Economic 
Evaluation  of  Alternatives  (EEoA). 

A  Critical  Evaluation  of  the  Status  Quo:  Two  Popular  Decision 
Criteria 

Today,  most  modern  military  investment  (and  disinvestment)  decisions  are  supported  by 
some  form  of  cost-benefit  analysis  (CBA).  The  US  Department  of  Defense  (DoD)  applies  CBA 
to  anything  from  milestone  decisions  for  Major  Defense  Acquisition  Programs  (MDAPs  and 
MAISs),  to  outsourcing  (OMB  Circular  A-76;  Eger  &  Wilsker,  2007),  to  public-private 
partnerships,  to  privatization,  to  Base  Realignment  and  Closure  (BRAC)  actions  (see  OMB 
Circular  A-94;  FAR;  DFARS;  DoD  5000  series,  etc.). 

When  benefits  cannot  be  expressed  in  monetary  terms,  analysts  develop  so-called 
“measures  of  effectiveness”  (MOEs),  in  which  case  CBA  is  generally  referred  to  as  “cost- 
effectiveness”  analysis  (OMB,  1992,  October  29). 15  The  most  common  methodology  and 
approach  for  building  MOEs  and  structuring  cost-effectiveness  analyses  is  alternately  referred 
to  as  Multiple-criteria  Decision-making  (MCDM),  Multi-attribute  Utility  Theory  (MAUT),  or 
Multiple-objective  Decision-making  (MODM)  (see  French,  1986;  Keeney  &  Raiffa,  1976; 

Clemen,  1996;  Kirkwood,  1997;  Parnell,  2006;  Ramesh  &  Zionts,  1997;  etc.). 

This  study  describes  some  limitations  of  the  current  decision  criteria  methodology  and 
proposes  an  alternate  methodology  derived  explicitly  from  a  constrained-optimization  approach, 
closer  in  spirit  to  the  economic  origins  of  cost-effectiveness  analysis  in  Gorman  (1 980);  Hitch 
and  McKean  (1967);  Michael  and  Becker  (1973);  Stigler  (1945);  Theil  (1952);  etc. — although 
often  attributed  to  Lancaster  (1969a;  1969b;  1971;  1979).  The  six  approaches  we  offer  to 
structure  an  Economic  Evaluation  of  Alternatives  (EEoA)  resolves  the  decision-criterion 


14  Appendix  3  (in  the  Mathematical  Appendix — available  upon  request)  reveals  the  static,  deterministic, 
multi-stage,  constrained-optimization,  micro-economic  production  (procurement  auction)  model  that 
underpins  the  central  EEoA  approach. 

15  Fisher  (1 965)  argues  that  “numerous  terms  [. . .]  convey  the  same  general  meaning  [. . .]  ‘cost-benefit 
analysis,’  ‘cost-effectiveness  analysis,’  ‘systems  analysis,’  ‘operations  analysis,’  etc.  Because  of  such 
terminological  confusion,  [...]  all  of  these  terms  are  rejected  and  ‘cost-utility  analysis’  is  employed  instead” 
(p.  185).  Although  this  study  uses  the  terms  “cost-benefit”  and  “cost-effectiveness”  interchangeably,  the 
assumption  throughout  is  that  neither  “benefits”  nor  “effectiveness”  can  be  measured  in  monetary  terms. 
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problem.  A  key  difference  between  the  MCDM  approach  to  an  AoA  and  the  Economic 
Evaluation  of  Alternatives  (EEoA)  is  that  instead  of  modeling  decision  alternatives  from 
competing  vendors  as  points  in  cost-effectiveness  space,  the  EEoA  models  the  alternatives  as 
functions  of  optimistic,  pessimistic,  and  most  likely  funding  (resource/budget)  scenarios. 

The  EEoA  approach  directly  responds  to  the  GAO’s  observation  that  affordability  needs 
to  be  an  integral  part  of  any  business  case  analysis  of  alternatives:  “[o]ur  work  in  [uncovering] 
best  practices  has  found  that  an  executable  business  case  [requires]  demonstrated  evidence 
that  [...]  the  chosen  concept  can  be  developed  and  produced  within  existing  resources 
[funding/budgets]”  (GAO,  2008,  p.  6).  Benchmarking  against  the  private  sector,  the  GAO 
emphasizes  that  “successful  commercial  enterprises  [...]  follow  a  disciplined  integrated  process 
during  which  the  pros  and  cons  of  competing  proposals  are  assessed  based  on  strategic 
objectives  [...]  and  available  resources  [budgets/funding]”  (GAO,  2009,  March  18,  p.  5, 
emphasis  added). 

A  distinctive  feature  of  defense  investment  decisions  is  that  multiple  criteria  such  as  cost 
and  effectiveness  cannot  easily  be  combined  into  a  single,  overall  objective  such  as 
“government  profitability.”  The  problem  of  ranking  public  investments  when  benefits  cannot  be 
expressed  in  dollars  has  spawned  an  extensive  literature  in  management  science,  operations 
research  and  the  decision  sciences. 

This  literature  models  investment  alternatives  as  bundles  of  measurable  characteristics 
(attributes  or  criteria).  Techniques  that  mostly  fall  under  the  umbrella  of  MCDM  are  routinely 
used  by  analysts  and  decision-makers  (for  example,  through  AoAs)  to  guide  public  investment 
decisions.  The  development  of  “Measures  of  Effectiveness”  (MOE’s)16  and  lifecycle  cost 
calculations  are  used  to  help  rank  alternatives.  An  ongoing  concern  is  how  to  integrate  costs 
and  effectiveness  in  the  final  selection  process  (see  Henry  &  Hogan,  1993;  Melese  &  Bonsper, 
1996,  December;  Melese,  Stroup,  &  Lowe,  1997;  etc.). 

In  their  pioneering  work  applying  economic  analysis  to  defense,  Hitch  and  McKean 
(1967)  define  a  “criterion”  as  the  “test  by  which  we  choose  one  alternative  [...]  rather  than 
another”  (p.  160).  They  stress  that  “[t]he  choice  of  an  appropriate  economic  criterion  is  [...]  the 
central  problem  in  designing  a  [cost-effectiveness]  analysis”  (p.  160). 

The  two  most  popular  decision  criteria  used  to  integrate  cost  and  effectiveness  in  AoAs 
are:  1)  to  construct  Benefit/Cost  (or  MOE/Cost)  ratios,  and  2)  to  assign  a  weight  on  cost  relative 
to  effectiveness  and  construct  a  weighted  average  of  cost  and  effectiveness  (often  using  a 
linear,  separable,  additive  “value”  function).  The  latter  decision  criterion  is  a  common 
prescription  for  AoAs  that  emerges  from  MCDM.  Both  approaches,  however,  are  problematic. 

We  first  focus  on  what  is  arguably  the  most  commonly  applied  criterion — Benefit/Cost 
ratios.  Then,  we  move  to  the  most  common  MCDM  decision  criterion — to  assign  a  relative 
weight  to  the  cost  (price)  of  alternatives  in  an  overall  value  function.  At  first  glance,  the 
Benefit/Cost  (MOE/Cost)  ratio  or  “bang-for-the-buck”  criterion  is  appealing.  However,  it  turns  out 


16  The  Defense  Acquisition  Guidebook,  Section  3.3.1 :  AoA  Plan,  states  that  “measures  of  effectiveness 
[...]  provide  the  details  that  allow  the  proficiency  of  each  alternative  in  performing  the  mission  tasks  to  be 
quantified  [. . .].  A  measure  of  performance  typically  is  a  quantitative  measure  of  a  system  characteristic 
(e.g.,  range,  [...]  logistics  footprint,  etc.)  chosen  to  enable  calculation  of  one  or  more  measures  of 
effectiveness”  (DoD,  2006,  July  7). 


DEFENSE  ACQUISITION  IN  TRANSITION 


- 13- 


to  be  largely  meaningless  unless  alternatives  are  constructed  for  a  specific  budget  scenario  or 
to  achieve  a  specific  level  of  effectiveness.  Meanwhile,  the  second  decision  criterion  can  also  be 
misleading  in  the  absence  of  a  specific  budget  scenario  (and  a  good  understanding  of 
“opportunity  costs”).17 


“Bang-for-the-Buck”  (Benefit/Cost  or  MOE/Cost)  Ratios 

It  is  relatively  well  known  that  a  Benefit/Cost  ratio  (or  “bang-for-the-buck”)  decision  criterion 
is  largely  meaningless  unless  alternatives  are  constructed  for  a  specific  budget  scenario  or  to 
achieve  a  specific  level  of  effectiveness.  Yet,  the  next  four  examples  illustrate  that  this  remains 
a  popular  decision  criterion,  even  when  alternatives  differ  in  both  costs  and  effectiveness. 

1 .  In  a  military  text  entitled  Executive  Decision  Making ,  the  author  offers  that  “[w]hen  we 
cannot  fix  cost  or  effectiveness,  we  might  combine  them  to  help  us  choose  between 
alternatives  [...].  If  neither  can  be  fixed  [...]  we  can  establish  a  cost/effectiveness 
ratio”  (Murray,  2002,  pp.  6-3,  6-10). 

2.  The  Department  of  the  Army’s  Economic  Analysis  Manual ,  in  a  section  entitled 
Comparing  Costs  and  Benefits,  states:  “When  the  results  yield  unequal  cost  and 
unequal  benefits  [...]  in  this  situation  all  alternatives  [...]  may  be  ranked  in 
decreasing  order  of  their  benefit/cost  ratios”  (DoA,  2001,  February,  p.  32). 

3.  Finally,  in  a  recent  landmark  RAND  study  on  Capabilities-based  Planning,  the  author 
falls  into  the  same  trap.  In  a  section  entitled  Choosing  Among  Options  in  a  Portfolio, 
Paul  Davis  (2002)  develops  “A  Notional  Scorecard  for  Assessing  Alternatives  in  a 
Portfolio  Framework,”  where  alternatives  differ  in  both  their  costs  and  effectiveness. 
Nevertheless,  the  decision  criterion  recommended  by  the  author  to  select  among 
alternative  options  in  “[t]he  last  column  is  the  ratio  of  effectiveness  over  cost”  (pp.  45- 
46). 


Each  of  these  diverse  examples  recommends  using  a  Benefit/Cost  ratio  as  the  decision 
criterion.  However,  another  RAND  analyst,  Gene  Fisher  (1971),  clearly  points  out  in  his  classic 
text  Cost  Considerations  in  Systems  Analysis: 

The  use  of  ratios  usually  poses  no  problem  as  long  as  the  analysis  is  conducted  in  [a] 
framework  [...]  with  the  level  of  effectiveness  or  cost  fixed.  However,  it  is  common  to 
encounter  studies  where  this  has  not  been  done,  with  the  result  that  the  comparisons 
[are]  essentially  meaningless,  (p.  11) 


17  Ironically,  given  a  budget  scenario,  there  is  no  need  to  take  the  MCDM  approach  that  underpins  most 
AoAs  since  it  is  possible  to  adopt  the  EEoA  approach — which  constructs  alternatives  to  fit  within  a  budget 
envelope,  converting  the  problem  into  a  straightforward  MOE  maximization. 
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Is  A1  really  superior  to  A2  ? 

LESSON:  DANGER  in  using  Benefit/Cost  (Bang/Buck)  or 
Cost/Benefit  (Buck/Bang)  ratios  without  anchoring  Budget  or  MOE 

▲  RANKING: 


MOE(Utils) 


m 

C/5 

(D 

Ch 

CD 

> 

•  i-H 

» 

O 

& 

Cm 

W 


Benefit/Cost  (A1)  >, Benefit/Cost  (A2) 

A2  O 


O 


Efficient  Set 


A1 

0  / 


'O 


Marginal  Benefit/Marqinal  Cost 

“The  perceived  benefits  of  the 
higher  priced  proposal  shall 
merit  the  additional  cost. . 

FAR  15.101-1(2)c 


Cost 


Figure  1.  Inappropriate  Application  of  Benefit/Cost  Ratios 

A  simple,  extreme  example  helps  illustrate  the  danger  in  using  Benefit/Cost  ratios 
without  anchoring  the  Budget,  or  a  specified  Measure  of  Effectiveness  (MOE).  Suppose 
Alternative  A1  in  Figure  1  costs  $10  million  and  yields  an  MOE  of  10  utils,  while  Alternative  A2 
costs  $1  billion  and  yields  an  MOE  of  900  utils.18  Applying  the  Benefit/Cost  ratio  criterion 
indicates  that  A1  has  a  bigger  “bang-for-the-buck”  since  it  returns  1  util  per  million  dollars,  while 
A2  only  offers  0.9  utils  per  million  dollars.  Strictly  using  Benefit/Cost  ratios  to  rank  alternatives  is 
dangerous  in  this  case  since  it  ignores  the  absolute  magnitude  of  the  costs  involved.  Suppose 
the  situation  was  reversed  and  A2  offered  a  higher  Benefit/Cost  ratio  than  A1 .  Anyone  that 
chooses  A2  strictly  on  the  basis  of  “bang-for-the-buck”  would  be  in  for  an  unpleasant  surprise  (a 
1 -billion  vs.  10-million-dollar  decision). 

Since  affordability  and  opportunity  costs  are  always  a  concern  in  public  investment 
decisions  (especially  those  made  through  the  PPBE  process,  requirements  generation  system, 
and  Defense  Acquisition  System),  it  is  imperative  that  analysts  and  decision-makers  explore  the 
budget  and  opportunity  cost  implications  of  going  with  the  high-cost  alternative  (for  example,  the 
extra  expenditure  of  $990  million  for  an  additional  890  utils  of  MOE)  or,  equivalently,  of  the 
savings  in  going  with  the  low-cost  alternative. 

In  applying  economic  analysis  to  defense,  Hitch  and  McKean  (1967)  warn: 

One  common  “compromise  criteria”  is  to  pick  that  [alternative]  which  has  the  highest 
ratio  of  effectiveness  to  cost.  [Maximizing  this  ration  is  the  [decision]  criterion.  [While]  it 
may  be  a  plausible  criterion  at  first  glance  [...]  it  allows  the  absolute  magnitude  of 


18  In  Figure  1 ,  the  slope  of  any  ray  from  the  origin  represents  a  constant  Benefit/Cost  ratio  anywhere 
along  that  ray.  The  steeper  the  slope,  the  greater  the  Benefit/Cost  ratio. 


DEFENSE  ACQUISITION  IN  TRANSITION 


-  15- 


[effectiveness]  or  cost  to  roam  at  will.  In  fact,  the  only  way  to  know  what  such  a  ratio 
really  means  is  to  tighten  the  constraint  until  either  a  single  budget  (or  particular  degree 
of  effectiveness)  is  specified.  And  at  that  juncture,  the  ratio  reduces  itself  to  the  test  of 
maximum  effectiveness  for  a  given  budget  (or  a  specified  effectiveness  at  minimum 
cost),  and  might  better  have  been  put  that  way  at  the  outset  [...].  The  test  of  maximum 
effectiveness  for  a  given  budget  (or  alternatively,  minimum  cost  of  achieving  a  specified 
level  of  effectiveness)  [...]  seems  much  less  likely  to  mislead  the  unwary.19  (pp.  165-167) 


Our  Economic  Evaluation  of  Alternatives  (EEoA)  approach  follows  this  and  another  of 
Hitch  and  McKean’s  (1967)  recommendations.  “As  a  starter  [...]  several  budget  sizes  can  be 
assumed.  If  the  same  [alternative]  is  preferred  for  all  [. . .]  budgets,  that  system  is  dominant  [. . .]. 
If  the  same  [alternative]  is  not  dominant  the  use  of  several  [...]  budgets  is  nevertheless  an 
essential  step,  because  it  provides  vital  information  to  the  decision  maker”  (p.  176). 

We  conclude  that  the  use  of  Benefit/Cost  ratios  as  a  decision  criterion  poses  no  problem 
as  long  as  the  analysis  is  structured  with  the  level  of  either  utility  (MOE)  or  budget/funding 
fixed.20  Since  Benefit/Cost  ratios  are  “misleading”21  in  any  context  in  which  alternatives  differ  in 
both  costs  (price)  and  benefits  (MOE),  decision  scientists  have  developed  another  decision 
criterion  to  rank  investment  options  in  AoAs.  This  second  popular  decision  criterion  is  examined 
below. 

Weighted  Averages  of  Cost  and  Effectiveness:  Assigning  a  Weight  to  Cost 


MCDM  is  often  used  as  an  umbrella  term,  and  we  will  do  so  here.  “In  the  literature  the 
terms  multi-attribute  decision  making  (MADM),  multi-criteria  decision  making  (MCDM),  and 
multi-objective  decision  making  (MODM)  are  used  almost  interchangeably”  (French,  1986,  p. 
105).  In  a  typical  MCDM  evaluation,  a  decision-maker  (DM)  is  asked  to  identify  desired 
attributes  (criteria/characteristics)  of  a  project,  program  or  system  to  fill  some  critical  capability 
gap,  given  a  specific  threat  scenario.  Next,  the  DM  is  asked  to  reveal  agreeable  trade-offs 
among  those  attributes.  An  exercise  of  this  sort  helps  analysts  uncover  the  DM’s  underlying 
trade-offs  or  “utility”  function,  used  to  generate  a  Measure  of  Effectiveness  (MOE)  for  each 
alternative.22 


19  The  authors  continue:  “Of  course,  if  the  ratios  did  not  alter  with  changes  in  the  scale  of  achievement  (or 
cost,  the  higher  ratio  would  indicate  the  preferred  system,  no  matter  what  the  scale  [...].  But  to  assume 
that  such  ratios  are  constant  is  inadmissible  some  of  the  time  and  hazardous  the  rest”  (Hitch  &  McKean, 
1967,  p.  167). 

20  An  additional  (necessary  and  sufficient)  condition  is  a  linear,  separable,  additive  objective  function. 

21  “Usually,  ratios  are  regarded  as  potentially  misleading  because  they  mask  important  information”  (DoD, 
2006,  July  7,  Section  3.3.1). 

22  “Measures  of  Effectiveness  [...]  provide  the  details  that  allow  the  proficiency  of  each  alternative  in 
performing  the  mission  tasks  to  be  quantified  [...].  A  measure  of  performance  typically  is  a  quantitative 
measure  of  a  system  characteristic  (e.g.,  range,  etc.)  chosen  to  enable  calculation  of  one  or  more 
measures  of  effectiveness  [...].  The  cost  analysis  normally  is  performed  in  parallel  with  the  operational 
effectiveness  analysis.  It  is  equal  in  importance  in  the  overall  AoA  process  [...].  [I]ts  results  are  later 
combined  with  the  operational  effectiveness  analysis  to  portray  cost-effectiveness  comparisons”  (DoD, 
2006,  July  7,  Section  3.3.1). 
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In  attempting  to  understand  a  DM’s  utility  function,  decision  scientists  beginning  with 
Saaty  (1977)  bridged  an  important  implementation  gap.  Objectives  (analytic)  hierarchy 
approaches  were  developed  that  help  reveal  underlying  utility  functions.  For  example,  an 
objectives  hierarchy  can  help  a  DM  work  down  from  a  high-level  objective  (provide  national 
security)  to  a  relevant  set  of  sub-objectives  (an  effective  airlift  capability),  to  specific  attributes 
(mobility,  transportability,  etc.),  and,  finally,  to  measurable  characteristics  (mobility=speed  (S), 
range  (R);  transportability=payload  (P),  weight  (W),  etc.). 


The  outcome  in  this  example  is  a  utility  function  for  airlift  capability:  U=U(M(S,R); 
T(P,W)),  where  the  characteristics  might  be  measured  respectively  in  mph,  miles,  cubic  feet, 
and  pounds.  The  standard  assumption  in  the  literature  is  to  define  a  linear,  separable  additive 
utility  function  that  generates  an  MOE  for  each  alternative  that  is  roughly  analogous  to  a 
weighted  average  of  its  attributes  (provided  certain  assumptions  are  satisfied  such  as  “additive 
independence,”  etc.  (see  French,  1986;  Keeney  &  Raiffa,  1976;  Keeney,  1994).  There  is  a  vast 
literature  concerned  with  eliciting  preference  weights  and  the  normalization  of  characteristics 
data  that  involves  several  important  issues  discussed  in  the  Mathematical  Appendix  (available 
upon  request). 

Temporarily  overlooking  these  issues,  it  is  interesting  to  note  in  passing  that  maximizing 
a  linear  multi-attribute  utility  function  subject  to  a  budget  constraint  yields  a  decision  rule 
analogous  to  the  Benefit/Cost  ratio  criterion  discussed  above.  Under  the  assumption  of  a  fixed 
budget  and  linear  additive  separable  utility  function,  the  Benefit/Cost  decision  rule  can  be  used 
to  evaluate  alternatives.  In  this  case,  the  winning  alternative  is  the  one  that  generates  the 
highest  MOE  per  dollar  or  the  biggest  “bang-for-the-buck.”  With  a  more  general  (non-linear) 
utility  function,  the  equivalent  optimization  generates  a  more  complex  Marginal  Benefit/Marginal 
Cost  decision  rule. 


In  reality,  the  MCDM  techniques  that  underpin  most  AoAs  often  do  not  rely  on  an  explicit 
discussion  of  the  budget  (affordability/funding/resources)  to  structure  the  decision  problem.  As  a 
consequence,  the  problem  is  generally  not  structured  as  a  constrained  optimization,  as 
described  above. 

Instead,  a  popular  decision-analysis  approach  is  to  simply  attach  a  weight  to  cost  and 
introduce  it  directly  into  the  utility  function.23  This  common  practice  generates  an  overall  “value” 
function  that  is  essentially  a  weighted  average  of  cost  and  effectiveness.  The  solution  is  found 
through  an  unconstrained  optimization  by  selecting  the  alternative  that  maximizes  the  “overall 
effectiveness”  or  “value”  function  V=V(MOE;  COST).  “Deterministic  decision  analysis  is 
concerned  with  finding  the  most  preferred  alternative  in  decision  space  by  constructing  a  value 
function  representing  a  decision  maker’s  preference  structure,  and  then  using  the  value  function 
to  identify  the  most  preferred  solution”  (Ramesh  &  Zionts,  1997,  p.  421). 


23  “In  the  European  Union,  a  legislative  package  intended  to  simplify  and  modernize  existing  public 
procurement  laws  was  recently  adopted.  As  before,  the  new  law  allows  for  two  different  award  criteria: 
lowest  cost  and  best  economic  value.  The  new  provisions  require  that  the  procurement  authority 
publishes  ex-ante  the  relative  weighting  of  each  criteria  used  when  best  economic  value  is  the  basis  for 
the  award”  (see  European  Commission,  2004a;  2004b). 
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The  linear,  additive  separable  version  of  this  value  function  is  often  used  to  calculate  a 
positively  weighted  MOE  and  negatively  weighted  cost  for  each  alternative.  For  example,  see 
Beil  &  Wein  (2003),  Che  (1993),  Clemen  (1996),  Kirkwood  (1997),  French  (1986),  Keeney  & 
Raiffa  (1976),  Keeney  (1994),  Hwang  and  Yoon  (1981),  Liberatore  (1987),  Pinker,  Samuel,  and 
Batcher  (1995),  Varzsonyi  (1995),  etc.  According  to  the  Defense  Acquisition  Guidebook :  “An 
AoA  is  an  analytical  comparison  of  the  operational  effectiveness,  suitability,  and  life-cycle  cost 
of  alternatives  that  satisfy  established  capability  needs.”  (DoD,  2006,  July  7,  Chapter  3.3) 

The  typical  decision  sciences’  approach  to  an  AoA  can  be  described  as: 

Given  several  Alternatives,  select  the  preferred  alternative  that  provides  the  Best  Value, 

or  Maximizes:  V(MOE,COST)  =  w1*MOE  -  w2*COST 

This  requires  two  important  modeling  efforts:  1)  MOE — Building  an  Effectiveness  model 
(non-cost  factors;  performance=quality,  schedule,  etc.);  and  2)  COST — Building  a  Cost_ model 
(costs/prices;  estimate  total  system  lifecycle  costs,  total  ownership  costs).  Once  the 
independent  modeling  efforts  are  completed,  the  overwhelming  challenge  is  to  assign  a  relative 
weight  to  cost  (w2  in  the  example  above).  A  typical  response  in  the  applied  literature  is  to  ask 
the  DM:  “How  important  is  cost  relative  to  effectiveness?” 

A  key  proponent  of  this  decision  methodology  offers  an  example  of  administrators  and 
regulators  asking  questions  such  as:  “Which  is  more  important,  costs  or  pollutant 
concentrations?”  (Keeney,  1994,  p.  797).  As  the  author  is  quick  to  point  out,  the  problem  with 
this  approach  is  that  without  some  estimate  of  the  total  budget  available  or  any  knowledge  of 
opportunity  costs  of  funds,  one  cannot  expect  the  DM  to  provide  a  sensible  answer.  In  fact,  the 
author  warns:  “I  personally  do  not  want  some  administrator  to  give  two  minutes  of  thought  to  the 
matter  and  state  that  pollutant  concentrations  are  three  times  as  important  as  cost”24  (Keeney, 
1994,  p.  797).  The  Federal  Acquisition  Regulations  (FAR)  and  the  Office  of  Management  and 
Budget  (OMB)  both  promote  similar  approaches:25 

■  “The  solicitation  shall  state  whether  all  evaluation  factors  other  than  cost/price,  when 
combined  [i.e. ,  MOE],  are  significantly  more  important  than,  approximately  equal  to, 
or  significantly  less  important  than  cost/price”  (General  Services  Administration, 

2005,  March,  Section  15.101-1(2)). 

■  “The  specific  weight  given  to  cost  or  price  shall  be  at  least  equal  to  all  other 
evaluation  factors  combined  unless  quantifiable  performance  measures  can  be  used 
to  assess  value  and  can  be  independently  evaluated”  (OMB,  2003,  p.  B-8). 

Consider  an  extreme  case.  If  we  suppose  that  affordability  is  not  an  issue,  then  funding 
is  not  an  issue  and  the  budget  is  not  binding,  making  costs  irrelevant.  In  this  case,  a  zero  weight 


24  Surprisingly,  the  author  has  continued  to  write  prolifically  in  this  field  and  continued  to  promote  this 
decision  criterion,  apparently  never  taking  the  time  to  reflect  back  on  these  key  observations. 

25  According  to  the  FAR,  “source  selection”  is  the  decision  process  used  in  competitive,  negotiated 
contracting  to  select  the  proposal  that  offers  the  “Best  Value”  to  the  government.  “In  different  types  of 
acquisition,  the  relative  importance  of  cost  or  price  may  vary”  (General  Services  Administration,  2005, 
Section  15.101).  In  describing  some  lessons  learned,  Gansler  (2003)  recommend:  Use  performance- 
based  contracting;  Do  not  list  tasks  [mix  of  inputs],  instead  state  results  sought  or  problems  to  be  solved 
[desired  attributes/characteristics  of  outputs/outcomes];  Choose  contractors  according  to  “Best  Value”;  in 
the  source  selection,  trade-off  performance  and  price  instead  of  simply  awarding  to  the  lowest  bidder  (p. 
15). 
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should  be  assigned  to  costs  and  the  analysis  of  alternatives  can  be  made  exclusively  on  the 
basis  of  MOEs. 

Thus,  any  weight  applied  to  costs  must  reflect  an  implicit  concern  about  affordability 
(budgets/funding  levels).  Figures  2  and  3  offer  an  illustration.  Figure  2  reflects  a  situation  in 
which  the  decision-maker  believes  costs  to  be  important  enough  (and  thus  assigns  a  sufficiently 
large  relative  weight,  w2,  to  cost)  that  the  preferred  alternative  is  A1  (the  low-cost  option).  The 
opposite  case  is  illustrated  in  Figure  3. 26  How  does  a  decision-maker  (DM)  decide  on 
appropriate  weights  to  assign  to  MOE  and  costs?  A  key  hypothesis  in  the  EEoA  is  that  if  a  DM 
pays  any  attention  to  costs  (i.e.,  places  any  weight  on  cost)  it  is  because  there  is  a  (implicit) 
budget  constraint  or  opportunity  cost  of  funds  for  the  program.  This  is  directly  related  to  our 
higher-level  affordability  discussions  in  Section  1  that  involved  requirements  generation 
(JCIDS),  the  Defense  Acquisition  System  (DAS),  and  PPBE.27 

The  irony,  as  Keeney  (1994)  rightly  observed,  is  that  to  assign  any  weight  to  costs 
requires  the  DM  to  have  some  understanding  of  the  budget  (funding/resources)  available  and 
an  appreciation  of  relevant  opportunity  costs.  But  if  this  information  is  known,  then  the  DM  has 
no  reason  to  take  the  MCDM  approach  and  assign  a  weight  on  costs  since  the  more  robust, 
constrained-optimization  (mathematical  programming)  EEoA  approach  becomes  available. 

In  fact,  it  is  relatively  straightforward  to  demonstrate  that  even  if  the  DM  had  perfect 
information  about  the  budget  (funding/affordability)  and  attempted  to  interpret  that  information 
through  a  weight  assigned  to  the  cost  (price)  of  alternatives  (as  illustrated  in  Figures  2  and  3), 
the  rankings  that  result  would  only  coincidentally  correspond  to  rankings  obtained  under  the  full 
information,  constrained-optimization  EEoA  (in  which  an  MOE  utility  function  is  maximized 
subject  to  the  budget  constraint).28 

This  is  a  damning  result  that  clearly  undermines  the  way  MCDM  is  typically  applied  to 
support  AoAs.  If  there  is  no  guarantee  this  MCDM  approach  will  yield  consistent  results  under 
full  information,  then  using  this  criterion  with  less  than  perfect  information  (i.e.,  without  explicit 
assumptions  about  affordability/budgets/funding),  is  clearly  problematic.  In  fact  GAO 
emphasizes  “[wjith  high  levels  of  uncertainty  [...]  funding  needs  are  often  understated”  (GAO, 
2008,  September  25,  p.  9). 


26  Note  that  the  slope  of  the  straight-line  indifference  curves  that  reflect  the  DM’s  relative  preference  (or 
trade-offs)  between  MOE  and  Cost  are  given  by  -w2/w1 . 

27  In  fact,  the  Army’s  Economic  Analysis  (EA)  Manual  states  that  “[a]  good  EA  should  go  beyond  the 
decision-making  process  and  become  an  integral  part  of  developing  requirements  in  the  PPBE  process” 
(DoA,  2001,  February,  p.  12). 

28  The  weight  on  cost  in  the  unconstrained-optimization  (MCDM)  approach  roughly  corresponds  to  the 
Lagrangian  multiplier  (shadow  price)  of  the  budget  constraint  in  the  constrained-optimization  (the  EEoA 
approach). 
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So  Which  Alternative  is  “Best”? 

Decision  Sciences  Approach 

Max  V  =  V(MOE,Cost)  =  wl  *MOE  -  w2*Cost 

Ask  Decision  Maker  What  is  More  Important:  MOE  or  Cost? 

(dMOE/dCost  =  w2/w1 ) 
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Figure  2.  When  Cost  is  Relatively  More  Important  than  Effectiveness 


Which  Alternative  is  “Best”? 

Decision  Sciences  Approach 

Max  V  =  V(MOE,Cost)  =  wl  *MOE  -  w2*Cost 


MOE(Utils) 


cn 

<D 

a 

<D 

> 
•  i-H 

O 

& 

w 


(dMOE/dCost  =  w2/w1) 


&A2 


,Q'  Al 


If  Performance  has  a 
sufficiently  greater  weight 

(w1»w2),  then  high  cost, 
high  effectiveness 
alternative  A2  wins 


Cost 


Figure  3.  When  Effectiveness  is  Relatively  More  Important  than  Cost 


In  conclusion,  the  popular  MCDM,  Decision  Sciences  approach  that  underpins  many 
AoAs  implicitly  attempts  to  capture  affordability  through  a  relative  weight  assigned  to  cost  in  a 
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value  function  such  as,  Maximize  V  =  V(MOE,Cost)  =  w1*M0E  -  w2*Cost. 29  Again,  to  quote 
Hitch  and  McKean  (1967): 

One  ubiquitous  source  of  confusion  is  the  attempt  to  maximize  gain  [w1*MOE]  while 
minimizing  cost  [w2*Cost]  [...]  If  a  person  approaches  a  problem  with  the  intention  of 
using  such  a  [decision]  criterion,  he  is  confused  to  begin  with  [...]  [A]  criterion  in  which 
the  budget  [. . .]  is  specified  has  the  virtue  of  being  aboveboard,  (pp.1 65-1 67) 

A  very  real  risk  in  this  MCDM  approach  is  that  if  AoAs  “fail  to  balance  needs  with 
resources  [funding/budgets],  [...]  un-executable  programs  [are  allowed]  to  move  forward,  [and] 
program  managers  [...]  are  handed  [...]  a  low  probability  of  success”  (GAO,  2009,  p.  10).  Rather 
than  attempt  to  get  a  DM  to  reveal  their  affordability  concerns  through  a  weight  assigned  to 
costs  (or  prices)  of  alternatives,  the  EEoA  recommends  a  more  transparent  and  accountable 
approach — to  treat  “cost  as  an  independent  variable”  (CAIV). 

The  CAIV  concept  adopted  here  follows  a  definition  posted  on  the  OUSD  (A&T)  website 
in  early  1999  that  CAIV  is  the  “DoD’s  acquisition  methodology  of  making  technical  and  schedule 
performance  a  function  of  available  budgeted  resources”  (see  Lorell  &  Graser,  2001,  p.  33). 

OMB  Circular  A-1 09  for  Major  Systems  Acquisition  mentions  the  goal  of  “design-to-cost”:  “Under 
the  CAIV  philosophy,  performance  and  schedule  are  considered  dependent  on  the  funds 
available  for  a  specific  program”  (OMB,  1976).  According  to  the  Defense  Acquisition  Guidebook, 
“all  participants  [...]  are  expected  to  recognize  the  reality  of  fiscal  constraints”  (DoD,  2006,  July 
7,  Section  3.2.4). 

Six  Ways  to  Structure  an  “Economic  Evaluation  of  Alternatives” 
(EEoA) 

We  have  identified  (what  we  believe  are  the  only)  six  ways  that  analysts  and  decision¬ 
makers  can  structure  a  deterministic  Economic  Evaluation  of  Alternatives  (EEoA)  that  avoid  the 
issues  (decision-criteria  problems)  discussed  in  the  last  section.  These  involve  two  main 
categories  of  approaches:  1)  Intra-program  analysis  and  2)  Inter-program  analysis.  The  first, 
third,  and  fourth  approaches  are  very  much  in  the  spirit  of  “cost  as  an  independent  variable” 
(CAIV).  By  recalling  the  earlier  quote  from  Hitch  and  McKean  (1967),  we  are  reminded  of  the 
first  two  EEoA  approaches:  “[A]  criterion  in  which  the  budget  or  level  of  effectiveness  specified 
has  the  virtue  of  being  aboveboard”  (p.  167).  The  six  EEoA  approaches  appear  in  Table  1 . 


29  In  a  section  describing  “Building  a  Model,”  Fisher  (1965)  comments:  “Since  by  definition  a  model  is  an 
abstraction  from  reality,  the  model  must  be  built  on  a  set  of  assumptions.  These  assumptions  must  be 
made  explicit.  If  they  are  not,  this  is  to  be  regarded  as  a  defect  of  the  model  design”  (p.  190).  It  is  easy  to 
conceal  the  importance  of  affordability  (budget/funding)  issues  in  the  MCDM,  Decision  Sciences 
approach  that  underpins  many  AoAs.  In  sharp  contrast,  the  Economic  Evaluation  of  Alternatives 
approach  encourages  explicit  affordability  (budget/funding)  assumptions. 
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Table  1.  Six  Approaches  to  Structure  an  EEoA 


I)  INTRA-PROGRAM  ANALYSIS 

A)  Build  Alternatives 

1 .  Fixed  Budget  Approach 

2.  Fixed  Effectiveness  Approach 

3.  Expansion  Path  Approach  (Construct  alternatives  as  Cost- 
output/Effectiveness  Relations  or  “Response  Functions”:  Multi-stage  Micro- 
economic  Production  Model) 

B)  Modify  Existing  Alternatives:  “Level  the  Playing  Field” 

4.  Modified  Budget  Approach:  GOTO  1. 

5.  Modified  Effectiveness  Approach:  GOTO  2. 

II)  INTER-PROGRAM  ANALYSIS 

6.  Opportunity  Cost/Benefit  Approach 

In  the  case  of  Intra-program  analysis,  the  decision-maker  (DM)  associated  with  the 
program  is  assumed  to  have  sufficient  information  to  be  able  to  select  an  alternative  without 
reference  to  competing  programs.  That  is  not  the  case  in  Inter-program  analysis,  which  requires 
an  explicit  “opportunity  cost  approach.” 

There  are  two  possibilities  highlighted  within  the  Intra-program  analysis  approach.  The 
first  possibility  is  when  DMs  (analysts)  are  able  to  construct/define/build  alternatives 
(“endogenous  alternatives”).  The  second  possibility  is  when  the  alternatives  are  already 
constructed/defined/built  and  must  simply  be  evaluated  (“exogenous  alternatives”). 

This  section  describes  each  of  the  six  Economic  Evaluation  of  Alternatives  (EEoA) 
approaches  in  some  detail.  The  Mathematical  Appendix  describes  the  static,  deterministic, 
multi-stage,  constrained-optimization,  micro-economic  production  (procurement  auction)  model 
that  underpins  the  third,  and  most  general,  approach  to  the  EEoA,  the  Expansion  Path 
Approach.  We  begin  with  the  Fixed  Budget  Approach,  based  on  the  earlier  quote  from  Hitch  and 
McKean  (1967):  “The  test  of  maximum  effectiveness  for  a  given  budget  seems  much  less  likely 
to  mislead  the  unwary”  (p.  167,  emphasis  added). 

1.  Fixed  Budget  Approach 

In  his  groundbreaking  book  Cost  Considerations  in  Systems  Analysis,  Fisher  (1971) 
states:  “In  the  fixed  budget  case,  the  alternatives  being  considered  are  compared  on  the  basis 
of  effectiveness  likely  to  be  attainable  for  the  specified  budget  level”  (p.  12).  In  other  words, 
Fisher  also  explains  that,  “The  analysis  attempts  to  determine  that  alternative  (or  feasible 
combination  [...])  which  is  likely  to  produce  the  highest  effectiveness”  (p.  10). 

In  a  footnote,  Fisher  (1971)  adds:  “the  fixed  budget  situation  is  somewhat  analogous  to 
the  economic  theory  of  consumer  [optimization...].  For  a  given  level  of  income  [budget]  the 
consumer  is  assumed  to  behave  in  such  a  way  that  he  maximizes  his  utility”  (p.  10).  Drawing  on 
this  comparison,  the  Fixed  Budget  Approach  to  the  EEoA  leverages  Lancaster’s  “characteristics 
approach  to  demand  theory”  (Lancaster,  1969a;  1969b;  1971;  1979).  Originating  in  the  works  of 
Gorman  (1980),  Stigler  (1945),  Theil  (1952),  and  others  (that  also  provided  an  early  foundation 
for  some  of  the  MCDM  literature),  Lancaster  offers  economists  (and  defense  analysts)  a  familiar 
way  to  analyze  the  consumer  (or  defense  DM’s)  choice  problem  (such  as  choosing  among 
defense  investment  alternatives). 
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In  Lancaster’s  model,  different  vendors  generate  different  bundles  of  characteristics 
evaluated  by  decision-makers  (“consumers”).  Lancaster’s  model  proposes  that  to  choose 
among  alternative  bundles  of  commodities  (say  computers),  defense  decision-makers  maximize 
their  utility  function,  defined  over  a  desired  set  of  criteria,  attributes,  or  characteristics — hence 
the  term  adopted  by  decision  scientists,  Multiple-criteria  Decision-making  (MCDM) — subject  to  a 
budget  [funding/affordability]  constraint,  which  is  mostly  not  adopted  by  decision  scientists.30  In 
this  approach,  the  cost-effective  alternative  is  the  one  that,  for  a  given  budget  or  expenditure, 
generates  the  best  mix  of  characteristics,  evaluated  using  the  decision-maker’s  utility  function. 

Cost-Effectiveness  EEoA 

Build  Alternatives 

1.  Fixed  Budget  Approach 

Maximize  Effectiveness  subject  to  Budget  Constraint 
(construct  alternatives  for  given  budget) 


Cost($) 


Outsourcing  Opportunity: 

Can  we  get  more  bang  for  the 
same  bucks? 


/Benefit/Cost 


Figure  4.  Fixed  Budget  Approach 

This  Fixed  Budget  Approach  is  the  first  of  six  ways  proposed  to  structure  an  Economic 
Evaluation  of  Alternatives  (EEoA)  and  is  illustrated  in  Figure  4. 31  The  Budget  estimate  for  the 
program  in  Figure  4  is  set  at  level  B*.  The  three  alternatives  constructed  given  this  budget  are 
A1 ,  A2,  and  A3.  Given  its  superior  performance  in  terms  of  its  MOE,  A3  wins  the  competition, 
which,  in  this  case,  can  also  be  determined  from  its  Benefit/Cost  ratio. 

2.  Fixed  Effectiveness  Approach 


30 

Note  that  we  refer  to  the  usual  deterministic  “utility  function”  that  is  conventional  in  the  economics 
literature.  This  is  in  contrast  to  the  way  a  utility  function  is  typically  defined  in  the  decision  sciences  and 
operations  management  literature  as  a  stochastic  function.  The  “value  function”  described  in  the  latter 
literature  is  similar  to  our  “utility  function,”  except  that  costs  can  enter  into  a  value  function  and  are 
excluded  from  our  utility  function  since  they  appear  as  part  of  the  budget  constraint. 

31  Note  that  in  the  first  and  second  EEoA  approaches,  since  either  the  budget  (funding  level)  or  MOE 
(level  of  effectiveness)  is  anchored  in  the  constrained  optimization,  the  Benefit/Cost  ratio  decision 
criterion  can  be  used  as  a  decision  rule  in  the  selection  process.  The  steeper  the  slope  from  the  origin 
through  an  alternative  (A1 ,  A2,  A3),  the  bigger  the  “bang-for-the-buck.” 
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The  second  way  to  structure  an  EEoA  is  the  dual  of  the  first:  minimize  the  cost  of 
achieving  a  given  MOE.  RAND  Corporation’s  AoA  for  the  KC-135  Recapitalization  adopts  this 
approach,  stating:  “in  this  AoA,  the  most  ‘cost-effective’  alternative  [fleet]  means  precisely  the 
alternative  whose  effectiveness  meets  the  aerial  refueling  requirement  at  the  lowest  cost’ 
(Kennedy  et  al.,  2006,  p.  7,  emphasis  added).  Figure  5  offers  an  illustration. 

Another  example  is  the  section  on  cost-effectiveness  analysis  in  OMB  Circular  A-94  that 
states:  “A  program  is  cost-effective  if,  on  the  basis  of  life  cycle  cost  analysis  of  competing 
alternatives,  it  is  determined  to  have  the  lowest  costs  [. . .]  for  a  given  amount  of  benefits  [. . .]. 
Cost-effectiveness  analysis  can  also  be  used  to  compare  programs  with  identical  costs 
[budgets/funding]  but  differing  benefits”  (OMB,  1992,  October  29,  p.  4).  The  latter  part  of  the 
quote  refers  to  the  first  approach  to  structuring  an  EEoA,  and  the  former  refers  to  the  second 
approach. 


Cost-Effectiveness  EEoA 

Build  Alternatives 


2.  Fixed  Effectiveness  Approach 

Dual:  Minimize  Costs  subject  to  Effectiveness  Constraint 
(construct  alternatives  for  given  MOE) 


:  Benefit/Cost 


Outsourcinq  Opportunity: 

Can  we  spend  less  bucks  for 
the  same  bang?  OMB 

Circular  A-76:  Statement  of 
Work(SOW);  Invitation  for  Bid(IFB) 

If  Al  is  the  In-house  MEO,  then 
“Outsourcing’s  Out  &  Insourcing’s  In” 

(DoD  FY  2005  :  71%  A-76  won  by  in-house  MEO) 

Cost($) 


Figure  5.  Fixed  Effectiveness  Approach 

Another  example  of  the  Fixed  Effectiveness  Approach  to  structuring  an  EEoA  is  given  by 
public-private  (competitive  sourcing)  competitions  conducted  under  OMB  Circular  A-76,  which 
“requires  [...]  a  structured  process  for  [evaluating]  the  most  efficient  and  cost-effective  method 
of  performance  for  commercial  activities”  (2003,  May  29).  This  involves  four  steps:  1)  develop  a 
Statement  of  Work  (SOW)  or  Performance  Work  Statement  (PWS)  to  define  desired 
performance/effectiveness,  2)  construct  the  Most  Efficient  Organization  (MEO)  for  the  in-house 
competitor,  3)  issue  an  Invitation  for  Bid  (IFB)  for  well-defined,  routine  commercial  activities 
(SOW  or  PWS),  and  4)  compare  bids  or  proposals  (source  selection)  and  select  the  “least  cost” 
for  IFB. 
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Finally,  Title  10,  Subtitle  A,  Part  IV,  Chapter  146,  Section  2462  of  the  US  Code  reads:  “A 
function  of  the  Department  of  Defense  [...]  may  not  be  converted  [...]  to  performance  by  a 
contractor  unless  the  conversion  is  based  on  the  results  of  a  public-private  competition  that  [...] 
examines  the  cost  of  performance  of  the  function  by  Department  of  Defense  civilian  employees 
and  the  cost  of  performance  of  the  function  by  one  or  more  contractors  to  demonstrate  whether 
converting  to  performance  by  a  contractor  will  result  in  savings  to  the  Government  over  the  life 
of  the  contract”  (2007,  January  3).  This  offers  another  example  of  the  Fixed  Effectiveness 
Approach  to  structuring  an  EEoA. 

3.  Expansion  Path  (Response  Function)  Approach 

Hitch  and  McKean  (1967)  strongly  hint  at  the  third  way  to  structure  an  EEoA:  “The  test  of 
maximum  effectiveness  fora  given  budget  seems  much  less  likely  to  mislead  the  unwary”  (p. 
167).  They  explain,  “As  a  starter,  [...]  several  budget  sizes  can  be  assumed.  If  the  same 
[alternative]  is  preferred  for  all  [...]  budgets,  that  system  is  dominant.  If  the  same  [alternative]  is 
not  dominant,  the  use  of  several  [...]  budgets  is  nevertheless  an  essential  step,  because  it 
provides  vital  information  to  the  decision  maker”  (p.  1 76,  emphasis  added). 

This  third  way  to  structure  an  EEoA  is  the  foundation  for  all  the  others  and  is 
described  mathematically  in  the  Mathematical  Appendix  (available  upon  request).  It  is 
modeled  as  a  three-step  process  that  involves  multiple  players. 

For  ease  of  exposition,  we  assume  three  players:  the  military  buyer  and  two 
private  vendors.  The  first  step  is  for  the  military  buyer  to  publish  a  synopsis  of  the 
solicitation.  This  synopsis  (solicitation)  states  all  significant  non-price  factors 
(criteria/attributes/characteristics)  that  the  agency  expects  to  consider  in  evaluating 
proposals,  along  with  optimistic,  pessimistic  and  most  likely  estimates  of  the  budget. 


ECONOMIC  APPROACH:  Endogenous  Alternatives  (“Engel  Curves”) 
3.  Expansion  Path  (Response  Function)  Approach 
(Alternatives  are  Cost-Effectiveness  Relations,  not  Points) 
Explore  impact  of  budget  cuts  (Identify  vendor  responses) 


Source  Selection  Decision:  A2  for  pessimistic  budget;  A1  for  optimistic  budget 
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Figure  6.  Expansion  Path  Optimization  Approach 


Assuming  the  award  will  be  made  without  discussions  (pursuant  to  FAR  52.212-1  and 
52.215-1  (General  Services  Administration,  2005)),  the  military  buyer  has  a  secret  scoring  rule32 
used  to  rank  vendors  that  is  only  revealed  after  the  award  of  the  contract.  Once  a  solicitation  is 
issued  in  the  form  of  an  RFP  or  IFB,  interested  vendors  submit  their  offers  and  the  selection 
process  begins.33 

Each  vendor  is  assumed  to  have  different  production  and  cost  functions  (to  generate  the 
attributes).  The  vendors  constrained  optimizations  define  distinct  expansion  paths,  one  for  each 
vendor.  From  the  Envelope  Theorem,  the  Lagrangian  multiplier  in  each  vendor’s  optimization 
reveals  the  marginal  product  (the  extra  output  or  attribute  mix  possible  for  them  to  produce) 
from  relaxing  the  funding  constraint,  i.e.,  using  a  more  optimistic  budget. 

Economic  Evaluation  of  Alternatives  Approach: 

Military  Buyer  Goal:  Select  an  alternative  that  Maximizes 
MOE  =  utility  function  =  U(non-cost  factors/attributes), 

Subject  to  BUDGET  constraint  =  TC 

Vendor  Goal:  Select  a  mix  of  non-cost  factors  that 

Maximizes  Q  =  Production  Function  =  Q(non-cost  factors/attributes) 

Subject  to  TC  =  Sum  of  Costs  of  Attributes  =  cl  x  al  +  c2  x  a2  +. . .  <=  Budget 
Military  Buyer: 

(MOE)  build-effectiveness  model  (non-cost  factors:  Performance  =  quality,  schedule, 
etc.) 

(COST)  build-cost  model  (costs/prices:  Estimate  total  system  lifecycle  costs,  total 
ownership  costs) 

(BUDGET)  Estimate  budget  (funding  level  for  the  program) 

Private  Vendor: 

(Q)  Understand  Production  Function  Generates  Attributes 
(TC)  Identify  cost  of  producing  each  attribute 

Construct  Alternatives  as  a  function  of  the  Military  Buyer’s  Budget  constraint 

This  EEoA  approach  illustrated  in  Figure  6  follows  Hitch  and  McKean’s  (1967) 
recommendations:  “As  a  starter  [...]  several  budget  sizes  can  be  assumed.  If  the  same 
[alternative]  is  preferred  for  all  [. . .]  budgets,  that  system  is  dominant  [. . .].  If  the  same 
[alternative]  is  not  dominant  the  use  of  several  [...]  budgets  is  nevertheless  an  essential  step, 
because  it  provides  vital  information  to  the  decision  maker”  (p.  176). 

The  expansion  path  for  each  vendor  (see  Mathematical  Appendix)  reveals  what  that 
vendor  can  offer  at  different  budget  levels  (e.g.,  pessimistic,  most  likely  and  optimistic).  The  set 
of  each  vendor’s  proposals  under  the  different  budget  scenarios  is  an  “alternative.”  When  the 
set  of  expansion  path  proposals  of  each  vendor  are  transformed  (through  the  government’s 


32  The  buyer  can  request  a  single  offer  from  each  supplier  and  choose  the  one  he  prefers  among  the 
submitted  offers.  “We  call  this  procedure  a  ‘single-bid  auction  with  secret  scoring  rule’”  (Asker  &  Cantillon, 
2004,  p.  1). 

33 

The  budget  announcements  are  analogous  to  an  agency  exploring  in  order  to  uncover  its  true 
“reservation  price”  for  the  acquisition  (given  the  competing  demands  for  scarce  budgets).  The  adoption  of 
this  approach  of  evaluating  vendor  proposals  under  different  reservation  prices  could  eventually  lead  to 
greater  use  of  fixed-price  contracts. 
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utility  function)  into  a  cost-utility  or  cost-effectiveness  (MOE)  function  for  that  particular  vendor, 
then,  given  a  range  of  likely  budgets  for  the  program,  the  most  effective  vendor  over  that  range 
of  budgets  can  be  selected  (see  Figure  6). 

This  approach  explicitly  addresses  a  key  concern  of  the  GAO  that  “A  cost  estimate  is 
[...]  usually  presented  to  decision  makers  as  a  [...]  point  estimate  that  is  expected  to  represent 
the  most  likely  cost  of  the  program  but  provides  no  information  about  the  range  of  risk  and 
uncertainty  or  level  of  confidence  associated  with  the  estimate”  (GAO,  2009,  p.  9).  The  three- 
stage  procurement  auction  process  is  summarized  in  Table  2  below. 


Table  2.  Three  Stage  Multi-attribute  Procurement  Auction  (Expansion  Path  Optimization 

Approach) 


1)  First  Stage:  (CAIV) 

-  The  DoD  provides  notional  budget  guidance  (B)  to  alternative  vendors  for  the 
program.  The  DoD  searches  for  the  optimum  product  (Procurement)  and/or  service 
(R&D;  O&M)  package  that  it  can  obtain  at  that  price,  B.  The  DoD  also  reveals 
optimistic  and  pessimistic  budget  guidance. 

-  The  DoD  defines  the  set  of  characteristics/attributes  that  it  values,  and  this  is 
known  to  vendors.  However,  the  DoD’s  precise  utility  function  over  those 
characteristics  is  unknown  to  vendors  (secret  scoring  rule). 

2)  Second  Stage:  (Target  Costing) 

-  Vendors  have  different  costs  and  production  functions  for  generating  products  or 
services  (defined  as  bundles  of  characteristics). 

-  Each  vendor  maximizes  its  output  offer  (an  optimal  mix  of  the  desired 
characteristics)  subject  to  its  particular  budget  constraint  (which  includes  the 
DoD’s  budget  guidance  and  the  vendor’s  individual  costs  to  produce  a  unit  of  each 
characteristic). 

-  This  is  the  product  and/or  service  package  (output)  a  particular  vendor  is  able 

to  propose  for  each  possible  budget  (B),  given  its  production  function  (technical 
production  possibilities)  and  its  costs  of  generating  those  characteristics. 

3)  Third  Stage:  (Selection) 

With  the  latest  budget  forecast,  the  DoD  selects  among  the  optimized  characteristic 
bundles  proposed  by  each  vendor  the  bundle/alternative  (total  product/service 
package)  that  maximizes  the  DoD’s  utility  function. 


Whereas  the  first  three  ways  to  structure  an  EEoA  assume  that  alternatives  can  be 
generated  by  the  decision-maker,  the  last  three  assume  that  alternatives  are  exogenously 
determined  and  that  the  decision-maker  must  choose  among  pre-specified  alternatives.  The 
interesting  cases  are  those  in  which  an  alternative  costs  more  but  offers  greater  utility,  while 
others  cost  less  and  offer  less  utility. 

4.  Modified  Budget  Approach 

Suppose  that  the  overall  budget  or  desired  level  of  effectiveness  for  a  program  is  not 
available  and  that  the  alternatives  are  derived  exogenously — for  example,  on  the  basis  of  a 
manpower  or  squadron  constraint  (e.g.,  one  computer  per  person  or  a  certain  number  of  aircraft 
per  squadron).  Then,  it  is  likely  that  the  pre-specified  alternatives  solicited  from  different  vendors 
have  different  costs  and  yield  different  measures  of  effectiveness  (MOE).  The  first  step  might  be 
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to  create  a  scatter  plot  of  effectiveness  versus  cost  (see  Figure  7).  In  the  absence  of  any  other 
information,  the  highest  cost  alternative  a  DM  is  willing  to  consider  can  be  used  as  a  notional 
budget  estimate  for  the  program. 


EEOA:  “LEVEL  THE  PLAYING  FIELD” 

4.  Modified  Budget  Approach  (GOTO  1  &  3) 


Modify  alternatives  to  equalize  budget 
(Identify  vendor  MOE  responses  to  budget  increase) 


Figure  7.  Modified  Budget  Approach 


The  fourth  way  to  structure  an  EEoA  recognizes  that  the  highest-cost  (highest-utility) 
alternative  under  consideration  (for  example,  A2  in  Figure  7)  reveals  a  possible  budget 
constraint.  By  “leveling  the  playing  field,”  the  decision-maker  asks  how  the  extra  money  might 
be  used  by  the  lower-cost  (lower-utility)  vendor  (A1)  to  increase  the  utility  of  that  alternative 
(from  A1  to  A1*).34  Note  that  this  effectively  returns  the  problem  to  the  first  (and  third)  way  of 
structuring  an  EEoA. 

5.  Modified  Effectiveness  Approach 

Similarly,  the  fifth  way  to  structure  an  EEoA  levels  the  playing  field  for  a  threshold  choice 
of  utility  (or  effectiveness),  returning  the  problem  to  the  second  (and  third)  way  of  structuring  an 
EEoA.  For  example,  in  Figure  8,  anchoring  the  desired  MOE  at  a  target  level  such  as  that 
offered  by  vendor  2,  the  government  would  return  to  vendor  1  and  ask,  how  much  would  it  cost 
to  achieve  the  same  target  level  of  MOE?  In  Figure  8,  vendor  1  is  preferred  since  the  response 
(A1=>A1*)  minimizes  the  budget  required. 


34  Alternatively,  different  valuable  uses  for  the  money  saved  by  choosing  the  lower-cost  alternative  could 
be  brought  into  the  effectiveness  calculation.  Some  will  recognize  this  search  for  the  “next  best  alternative 
use  of  funds”  as  the  standard  economic  definition  of  opportunity  costs.  This  sets  the  stage  for  the  sixth 
way  to  structure  an  EEoA. 
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EOA:  “LEVEL  THE  PLAYING  FIELD” 

5.  Modified  Effectiveness  Approach  (GOTO  2  &  3) 
Modify  alternatives  to  equalize  MOE 
(Identify  vendor  COST  responses  to  higher  MOE  requirement) 


Figure  8.  Modified  Effectiveness  Approach 


6.  Opportunity  Cost  (or  Effectiveness)  Approach 

Finally,  what  if  1)  we  cannot  modify  alternatives  to  obtain  response  functions,  and  2)  we 
don’t  know,  or  cannot  assume,  a  given  budget  or  desired  level  of  MOE?  In  this  case,  some 
alternatives  (bundles)  cost  more  but  offer  more  effectiveness,  while  others  cost  less  and  offer 
less  effectiveness  (“efficient  set”).  The  sixth  and  final  way  to  structure  an  EEoA  involves  an 
inter-program  comparison  we  call  the  Opportunity  Cost  Approach. 

Rather  than  modify  the  alternatives  to  level  the  playing  field,  the  Opportunity  Cost 
Approach  accepts  both  lower-cost,  lower-effectiveness  alternatives  (A1  in  Figure  9)  and  higher- 
cost,  higher-effectiveness  alternatives  (A2)  but  requires  a  more  challenging  inter-program 
analysis. 

The  main  challenge  in  selecting  an  alternative  in  this  context  is  that  the  DM  must  reach 
beyond  the  immediate  program,  A,  into  higher-level  inter-program  considerations  (perhaps 
entering  the  requirements  generation  system  or  the  PPBE  process). 

If  the  alternatives  are  exogenously  determined,  and  it  is  not  possible  to  level  the  playing 
field,  then  to  find  the  most  cost-effective  solution  requires  information  about  other  competing 
programs  (e.g.,  program  B  in  Figure  9).  “[T]he  assessment  should  provide  details  as  to  how 
excess  funding  [...]  demands  will  be  accommodated  by  reductions  in  other  mission  areas,  or  in 
other  [...]  accounts”  (DoD,  2006,  July  7,  Section  3.2.2). 

This  involves  an  inter-program  analysis  similar  to  that  illustrated  in  Figure  9.  What  is  the 
loss  in  utility  in  other  programs  that  might  be  sacrificed  (B2=>B1)  for  the  funds  to  be  made 
available  to  purchase  greater  utility  in  the  program  under  review  (A1=>A2)?  Alternatively,  how 
much  more  utility  might  the  extra  money  generate  somewhere  else  if  we  went  with  the  low-cost 
alternative  (A1)?  These  are  tough  but  useful  questions  that  break  through  the  sub-optimization 
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of  most  traditional  AoAs.  In  this  way,  the  EEoA  approach  encourages  critical  communication  to 
take  place  between  different  layers  of  the  organization.35 


6.  Opportunity  Cost  Approach 

(INTER-PROGRAM  Marginal  Analysis) 

A)  Question:  Where  is  the  extra  money  coming  from  if  I 
buy  the  high  cost  alternative? 

B)  Question:  Where  is  the  extra  money  going  if  I  buy  the 
low  cost  alternative? 


MOEi(Utils) 


Program  A 


MOEi(Utils) 


Program  B 


A2 

o 


B2 

o 


A1 

o 


B1 

o 


Cost($)  Cost($) 

Figure  9.  Opportunity  Cost  Approach 


The  bottom  line  is  that  it  is  often  more  transparent,  efficient,  and  effective  to  develop 
MOEs  that  are  independent  of  costs  and  to  treat  costs  as  an  independent  variable  (CAIV). 
Equally  important  are  the  roles  of  budget  (funding)  forecasts  and  opportunity  costs  in  helping 
structure  defense  investment  decisions.  Structuring  an  Economic  Evaluation  of  Alternatives 
(EEoA)  using  one  of  the  six  approaches  summarized  in  Table  1  could  help  achieve  the  primary 
goal  of  defense  acquisition  reform — to  help  coordinate  the  requirements  generation  system 
(JCIDS),  Defense  Acquisition  System  (DAS),  and  PPBE  to  lower  the  costs  of  defense 
investments  and  improve  performance  and  schedules. 

Conclusion:  A  Decision  Map  for  Decision-makers 

This  study  identified  several  major  challenges  that  face  current  military  cost- 
effectiveness  analyses.  It  also  critically  examined  key  assumptions  of  the  decision  sciences’ 


35  Fisher  (1965)  quotes  Secretary  of  Defense  Robert  McNamara:  “Suppose  we  have  two  tactical  aircraft 
which  are  identical  in  every  important  measure  of  performance  [MOE]  except  one — aircraft  A  can  fly  ten 
miles  per  hour  faster  than  Aircraft  B.  Thus,  if  we  need  about  1,000  aircraft,  the  total  additional  cost  would 
be  $1  Omillion.  If  we  approach  this  problem  from  the  viewpoint  of  a  given  amount  of  resources,  the 
additional  combat  effectiveness  [...]  of  Aircraft  A  would  have  to  be  weighed  against  the  additional  combat 
effectiveness  which  the  same  $1  Omillion  could  produce  if  applied  to  other  defense  purposes — more 
Aircraft  B,  more  or  better  aircraft  munitions,  or  more  ships,  or  even  more  military  family  housing  [. . .]  This 
kind  of  determination  is  the  heart  of  the  planning-programming-budgeting  [...]  problem  with  the  Defense 
Department.”  (p.182) 
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literature,  which  are  frequently  used  by  the  military  to  structure  acquisition  decisions.  An 
alternative  micro-economic  set  of  approaches  to  structure  acquisition  decisions  was  proposed, 
called  the  Economic  Evaluation  of  Alternatives  (EEoA). 

This  study  points  to  a  significant  weakness  in  the  multiple-criteria,  decision-making 
(MCDM)  approach  that  underpins  many  contemporary  AoAs.  The  weakness  is  that  while  MCDM 
techniques,  and  therefore  most  AoAs,  correctly  focus  on  lifecycle  costs  and  the  operational 
effectiveness  of  individual  alternatives,  affordability  is  often  only  implicitly  addressed  through  a 
weight  assigned  to  costs. 

In  contrast,  the  EEoA  approach  recommended  in  this  study  encourages  analysts  and 
decision-makers  to  include  affordability  explicitly  in  the  AoA.  This  requires  working  with  vendors 
to  build  alternatives  based  on  different  funding  (budget/affordability)  scenarios.  Supported  by  a 
static,  deterministic,  multi-stage,  constrained-optimization,  micro-economic  production 
(procurement  auction)  model  described  in  Section  3,  this  EEoA  approach  explicitly  addresses 
affordability  up-front.  The  key  difference  between  the  MCDM  approach  to  AoA,  and  the  EEoA 
approach,  is  that  instead  of  modeling  decision  alternatives  from  competing  vendors  as  points  in 
cost-effectiveness  space,  the  EEoA  models  alternatives  as  functions  of  optimistic,  pessimistic 
and  most  likely  funding  (resource/budget)  scenarios. 

The  primary  goal  of  this  study  was  to  help  improve  public  investment  decisions  by 
providing  a  set  of  six  approaches  practitioners  (acquisition  officials  and  others)  can  employ  to 
structure  an  Economic  Evaluation  of  Alternatives  (EEoA).  An  important  secondary  goal  of  the 
study  was  to  develop  a  Decision  Map  to  guide  practitioners  and  acquisition  officials  in 
structuring  cost-effectiveness  analyses  to  improve  defense  acquisition  outcomes.  The  Decision 
Map  to  structure  an  EEoA  appears  below. 
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Decision  Map  to  Structure  an  Economic  Evaluation  of  Alternatives  (EEoA) 


Figure  10.  Decision  Map  to  Structure  EEoAs 
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T.  Gaul  (eds.),  MCDM:  Theory  and  applications.  New  York:  Springer-Verlag. 
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2003  -  2009  Sponsored  Research  Topics 

Acquisition  Management 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  BCA:  Contractor  vs.  Organic  Growth 

■  Defense  Industry  Consolidation 

■  EU-US  Defense  Industrial  Relationships 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

■  Managing  Services  Supply  Chain 

■  MOSA  Contracting  Implications 

■  Portfolio  Optimization  via  KVA  +  RO 

■  Private  Military  Sector 

■  Software  Requirements  for  OA 

■  Spiral  Development 

■  Strategy  for  Defense  Acquisition  Research 

■  The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 
Contract  Management 

■  Commodity  Sourcing  Strategies 

■  Contracting  Government  Procurement  Functions 

■  Contractors  in  21st  Century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Model  for  Optimizing  Contingency  Contracting  Planning  and  Execution 

■  Navy  Contract  Writing  Guide 

■  Past  Performance  in  Source  Selection 

■  Strategic  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  USAF  Energy  Savings  Performance  Contracts 

■  USAF  IT  Commodity  Council 

■  USMC  Contingency  Contracting 
Financial  Management 

■  Acquisitions  via  leasing:  MPS  case 

■  Budget  Scoring 

■  Budgeting  for  Capabilities-based  Planning 

■  Capital  Budgeting  for  DoD 
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■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Financing  DoD  Budget  via  PPPs 

■  Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition  Budgeting 
Reform 

■  PPPs  and  Government  Financing 

■  ROI  of  Information  Warfare  Systems 

■  Special  Termination  Liability  in  MDAPs 

■  Strategic  Sourcing 

■  Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 
Human  Resources 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 

■  Learning  Management  Systems 

■  Moral  Conduct  Waivers  and  First-tern  Attrition 

■  Retention 

■  The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 

■  Tuition  Assistance 
Logistics  Management 

■  Analysis  of  LAV  Depot  Maintenance 

■  Army  LOG  MOD 

■  ASDS  Product  Support  Analysis 

■  Cold-chain  Logistics 

■  Contractors  Supporting  Military  Operations 

■  Diffusion/Variability  on  Vendor  Performance  Evaluation 

■  Evolutionary  Acquisition 

■  Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 

■  Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

■  Optimizing  CIWS  Lifecycle  Support  (LCS) 

■  Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance  Activity 

■  Pallet  Management  System 

■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

■  RFID  (6) 

■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  Aegis  Microwave  Power  Tubes 


DEFENSE  ACQUISITION  IN  TRANSITION 


■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 
Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  Aegis  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our  website: 
www.acquisitionresearch.org 
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ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 

555  DYER  ROAD,  INGERSOLL  HALL 

MONTEREY,  CALIFORNIA  93943 


www.acquisitionresearch.org 


Defense  Acquisition  in  Transition 

6th  Annual  Acquisition  Research  Symposium 

—  m-*  \  rm 

The  Economic  Evaluation  of 
Alternatives  (EEoA) 

Dr.  Francois  Melese,  Professor  of  Economics 
Defense  Resources  Management  Institute  (DRMI), 

Naval  Postgraduate  School 


Analysis  of  Alternatives  (AoA) 

www.deskbook.osd.mil 


“An  AoA  is  an  analytical 
Comparison  of  the 
operational 
Effectiveness, 
suitability,  and 
Life-Cycle  Cost  of 
Alternatives  that  satisfy 
established 
Capability  needs.” 

Defense  Acquisition  GuideBook  Section  3.3) 


Source  Selection 

•  “Source  selection”  is  the  decision  process  used  in 
competitive,  negotiated,  contracting  to  select  the 
proposal  that  offers  the  Best  Value  to  the 
government.” 

•  In  the  UK  =>  “Best  value”  =  “Value  for  Money” 

•  In  the  US  =>  “In  different  types  of  acquisitions,  the 
relative  importance  of  cost  or  price  may  vary...” 

FAR  15.101 

•  “This  process  permits  tradeoffs  among  cost/price 
and  non-cost  factors  and  allows  the  Government  to 
accept  other  than  the  lowest  priced  proposal.” 


FAR  15.101-1  (2)c 


Analysis  of  Alternatives  (AoA) 

•  Decision  Sciences  (MCDM)  Approach: 

-  Objective:  Given  Alternatives,  Select  one  that 
Maximizes  Best  Value  =  V(MOE,COST) 

=  w1*MOE-  w2*COST 

•  “In  the  literature  the  terms  multi-attribute  decision  making  (MADM),  multi-criteria  decision  making 

(MCDM),  and  multi-objective  decision  making  (MODM)  are  used  almost  interchangeably.” 

S.  French  (1986)  Decision  Theory,  p.105 

•  (MOE)  Build  Effectiveness  model  (non-cost  factors: 
Performance=quality,  schedule,  etc.) 

•  Analytical  Hierarchy 

•  (COST)  Build  Cost  model  (costs/prices) 

•  Estimate  total  system  life  cycle  costs  (total  ownership  costs) 


Evaluation  of  Alternatives  (EEOA) 

•  Economics  Approach: 

-  Objective:  Select  Alternative  that 

Maximizes  MOE  =  Utility  =  U(non-cost  factors), 

Subject  to  AFFORDABILITY  (Funding/Budget)  constraint 


•  (MOE)  Build  Effectiveness  model  (non-cost 
factors:  Performance  =  quality,  schedule,  etc.) 

•  Analytical  Hierarchy 

•  (COST)  Build  Cost  model  (costs/prices) 

•  Estimate  total  system  life  cycle  costs  (total  ownership  costs) 

•  (FUNDING)  Estimate  budget  authority  (constraint) 

•  Construct  Alternatives 

•  In  the  Spirit  of:  Cost  as  an  Independent  Variable  (CAIV)  and 
Target  Costing 


Economic  Evaluation  of  Alternatives  (EEoA) 
Approaches:  Six  Wavs  to  Structure  an  EoA 

Build  Alternatives:  “Intra-Program  Analysis” 

1.  Fixed  Budget  Approach 

2.  Fixed  Effectiveness  Approach 

3.  Expansion  Path  Approach  (Construct  alternatives  as 
Cost-Output/Effectiveness  “Response  Functions”) 


Modify  Existing  Alternatives:  “Level  the  Playing  Field” 

4.  Modified  Budget  Approach:  GOTO  1. 

5.  Modified  Effectiveness  Approach:  GOTO  2. 

Cannot  Modify  Existing  Alternatives:  “Inter-Program  Analysis” 

6.  Opportunity  Cost/Benefit  Approach 


Analysis  of  Alternatives 


“[The]  two  basic  conceptual 
approaches  for  making  comparisons 
in  systems  analysis  [are]: 

1  .the  fixed  budget  approach,  and 
2.the  fixed  effectiveness  approach.” 


Fisher,  1970  (p.  78) 


Cost-Effectiveness  EEoA 

Build  Alternatives 


1.  Fixed  Budget  Approach 

Maximize  Effectiveness  subject  to  Budget  Constraint 
(construct  alternatives  for  given  budget) 


Can  we  get  more  bang  for  the 
same  bucks? 


Benefit/Cost 


Cost-Effectiveness  EEoA 

Build  Alternatives 


2.  Fixed  Effectiveness  Approach 

Dual:  Minimize  Costs  subject  to  Effectiveness  Constraint 

(construct  alternatives  forgiven  MOE) 


Benefit/Cost 


Cost($) 


Can  we  spend  less  bucks  for 
the  same  bang? 


AoA 


•  “Typically,  the  last  analytical  section  of  the  AoA  plan 
deals  with  the  planned  approach  for  the  cost- 
effectiveness  comparisons  of  the  study  alternatives.” 

THIS  IS  FIRST  STEP  OF  Economic  Evaluation  of  Alternatives 

•  “Cost  effectiveness  comparisons  in  theory  would  be 
simplified  if... all  the  alternatives  have  equal  effectiveness  (the 
best  alternative  is  the  one  with  the  lowest  cost)  or  equal  cost 
(the  best  alternative  is  the  one  with  the  greatest  effectiveness).” 

•  “In  actual  practice,  the  ideal  of  equal  effectiveness  or  equal 
cost  alternatives  is  difficult... to  achieve...” 

•  “A  common... comparison  is  a  scatter  plot  of  effectiveness 
versus  cost.” 


Defense  Acquisition  Guidebook  Section  3.3  www.deskbook.osd.mil 


Decision  Sciences  Approach 

Identify  “Efficient  Set” 

Weed  out  dominated  alternatives 


MOE(Utils) 


► 


Cost 


Decision  Sciences  Approach 

Those  remaining  form  an  efficient  set 
Which  Alternative  is  the  Best? 


MOE(Utils) 


G n 
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<D 

<D 

> 

• 
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A1 

o 


o 
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A2 


o 


Efficient  Set 


o 


Cost 


Source  Selection 

“The  solicitation  shall  state  whether  all  evaluation 
factors  other  than  cost/price,  when  combined 
[MOE],  are  significantly  more  important  than, 
approximately  equal  to,  or  significantly  less 
important  than  cost/price.” 


FAR  15.101-1(2) 

“[AJgencies  must:  a)  identify  the  specific  weight 
given  to  each  evaluation  factor..., and  b)  make  the 
specific  weight  for  cost  or  price  at  least  equal 

to  all  other  evaluation  factors  combined. . .” 


OMB  (2006)  “Competitive  Sourcing,”  Executive  Office  of  the  President,  p.21 


So  Which  Alternative  is  “Best”? 

Decision  Sciences  Approach 

Max  V  =  V(MOE,Cost)  =  w1*MOE  -  w2*Cost 

Ask  Decision  Maker  What  is  More  Important:  MOE  or  Cost? 

(d  MO  E/d  Cost  =  w2/w1 ) 


Cost 


Which  Alternative  is  “Best”? 

Decision  Sciences  Approach 

Max  V  =  V(MOE,Cost)  =  w1*MOE  -  w2*Cost 


MOE(Utils) 


G n 
G n 
<D 

<D 

> 

• 

o 

w 


(dMOE/dCost  =  w2/w1) 


A2 


If  Performance  has  a 
sufficiently  greater  weight 

(w1»w2),  then  high  cost, 
high  effectiveness 
alternative  A2  wins 


Cost 


“Most  Common  Critical  Mistake” 


“One  mistake  is  very  commonly  made  in  constructing 
value  models... illustrated  in  the  context  of... air 
pollution.  ..[i.e.  reducing  pollutant  concentrations] 

I  personally  do  not  want  some  administrator  to 
give  two  minutes  of  thought  to  the  matter  and 
state  that  [reducing]  pollutant  concentrations  [is] 
three  times  as  important  as  cost  ” 


DECISION  SCIENCES  EXPERT 

R.  Keeney  (1994)  “Using  Values  in  Operations  Research,”  Ops.  Research  42/5  p.797 


Question:  How  does  Decision  Maker  (DM)  decide  relative 
weights  to  assign  to  MOE  &  COSTS?  What  does  this  mean? 

•  Economist’s  Hypothesis: 

-  If  DM  cares  about  COSTS  (i.e.  places  any  weight  on 
cost)  it  is  because  there  is  a  budget  constraint  or 
opportunity  cost  of  obtaining  funds  to  pay  for  the  extra 
MOE  in  this  program. 

-  Otherwise  “Go  for  the  Gusto  (greatest  MOE)!” 

-  Decision  Sciences  (AoA)  approach  addresses  this  indirectly: 

Typical  Objectives  Hierarchy  includes  both  Max  MOE  &  Min  Costs  => 

Max  V  =  V(MOE,Cost)  =  w1*MOE  -  w2*Cost 

-  One  ubiquitous  source  of  confusion  is  the  attempt  to 
maximize  gain  while  minimizing  cost... if  a  person 
approaches  a  problem 
[decision]  criterion,  he 


with  the  intention  of  using  such  a 
is  confused  to  begin  with...”  (p.167) 

Hitch  C.  &  R.  McKean  1967  “The  Economics  of  Defense  in  the 
Nuclear  Age”  Harvard  University  Press,  Cambridge,  Mass. 


EEoA  Proposal:  Three-Step  Optimization 

“Tell  them  WHAT  you  want  and  roughly  what  you  can  afford,  then  let  them  figure  out  HOW  to  do  it.” 

I)  First  Stage:  (CAIV) 

-  DoD  provides  notional  budget  guidance  (B)  to  alternative  vendors  for  the 
program.  DoD  searches  for  the  optimum  product  (Procurement)  and/or 
service  (R&D;  O&M)  package  it  can  obtain  at  that  price,  B.  DoD  also  reveals 
optimistic,  most  likely,  and  pessimistic  budget  guidance. 

-  DoD  defines  the  set  of  characteristics/attributes  it  values  and  this  is 
known  to  vendors,  but  DoD’s  precise  Utility  Function  over  those 
characteristics  is  unknown  to  vendors. 


II)  Second  Stage:  (Target  Costing) 

-  Vendors  have  different  costs  and  production  functions  for  generating 
products  or  services  (defined  as  bundles  of  characteristics). 

-  Each  vendor  maximizes  its  output  offer  (an  optimal  mix  of  the  desired 
characteristics)  subject  to  their  particular  budget  constraint  (which 
includes  DoD’s  budget  guidance  and  the  vendor’s  individual  costs  to  produce 
a  unit  of  each  characteristic). 

-  This  is  the  product  and/or  service  package  (output)  a  particular  vendor 
is  able  to  propose  for  each  possible  budget  (B),  given  their  production 
function  (technical  production  possibilities)  and  their  costs  of  generating  those 
characteristics. 

III)  Third  Stage:  With  the  latest  budget  forecast,  DoD  selects  among 
optimized  characteristic  bundles  proposed  by  each  vendor,  the 
bundle/alternative  (system)  that  maximizes  DoD’s  Utility  Function. 


EEoA:  Vendor  Expansion  Paths  with  same  Costs 

Maximize  Attribute  Bundle  subject  to  Budget  Constraint 
(Assumption:  Two  Vendors  &  Two  Attributes) 


EEoA:  Vendor  Expansion  Paths  with  same  Technolo 
Maximize  Attribute  Bundle  subject  to  Budget  Constraint 


EEoA:  Procurement  Agency  Choice 

Maximize  Utility  subject  to  Budget  Authority  Constraint 


EEoA:  Procurement  Agency  Choice 

Maximize  Utility  subject  to  Budget  Authority  Constraint 


Economic  Evaluation  of  Alternatives 


Cost-Effectiveness  (Budget-Utility)  Analysis 
Vendor  Response  Functions  (Not  Points) 


What  if  we  cannot  build  Alternatives? 

(Alternatives  have  already  been  identified) 

•  “In  many  cases,  there  will  be  a  minimum  set  of 
alternatives  required  by  the  initial  analysis 
guidance.” 


•  In  most  AoAs,... comparisons  involve  alternatives 
that  have  both  different  effectiveness  and  cost, 
which  leads  to  the  question  of  how  to  judge  when 
additional  effectiveness  is  worth  additional  cost. 


(Defense  Acquisition  GuideBook  Section  3.3 
www.deskbook.osd.mil) 


EOA:  “LEVEL  THE  PLAYING  FIELD” 

4.  Modified  Budget  Approach  (GOTO  1  &  3) 
Modify  alternatives  to  equalize  budget 


(Identify  vendor  MOE  responses  to  budget  increase) 


EOA:  “LEVEL  THE  PLAYING  FIELD” 


5.  Modified  Effectiveness  Approach  (GOTO  2  &  3) 
Modify  alternatives  to  equalize  MOE 
(Identify  vendor  COST  responses  to  higher  MOE  requirement) 


6.  Opportunity  Cost  Approach 

(INTER-PROGRAM  Analysis) 

What  if 

-We  Cannot  Modify  alternatives  to  obtain 
response  functions? 

and 

-We  don’t  know  or  cannot  assume  a  given 
Budget  or  desired  MOE. 

Then  some  alternatives  (bundles)  cost  more  but 
offer  more  effectiveness,  while  others  cost  less 
and  offer  less  effectiveness  (“efficient  set”). 


6.  Opportunity  Cost  Approach 

(INTER-PROGRAM  Marginal  Analysis) 

A)  Question:  Where  is  the  extra  money  coming  from  if  I 
buy  the  high  cost  alternative? 

B)  Question:  Where  is  the  extra  money  going  if  I  buy  the 
low  cost  alternative? 


Enjoy  your  stay  in  Monterey! 


Defense  Resources  Management  Institute  (DRMI) 

Naval  Postgraduate  School 
www.nps.navy.mil/drmi 


Evaluation  of  Alternatives  (EEoA)  Proposal: 

Six  Wavs  to  Structure  an  EEoA 

Build  Alternatives:  “Intra-Program  Analysis” 

1.  Fixed  Budget  Approach 

2.  Fixed  Effectiveness  Approach 

3.  Expansion  Path  Approach  (Construct  alternatives  as  Cost- 
Output/Effectiveness  Relations  or  “Response  Functions”) 

Modify  Existing  Alternatives:  “Level  the  Playing  Field” 

4.  Modified  Budget  Approach:  GOTO  1. 

5.  Modified  Effectiveness  Approach:  GOTO  2. 

Cannot  Modify  Existing  Alternatives:  “Inter-Program  Analysis” 

6.  Opportunity  Cost/Benefit  Approach 


Economics  EOA  Approach 

Evaluation  of  Alternatives  (EOA): 

-  “[A]  criterion  in  which  the  budget  or  level  of  effectiveness  is 
specified  has  the  virtue  of  being  aboveboard.”  (p.  167) 
[EoA  approaches  1 .  and  2.] 


-  “The  test  of  maximum  effectiveness  for  a  given  budget 
seems  much  less  likely  to  mislead  the  unwary... ”(p.  167) 

-  “As  a  starter,... several  budget  sizes  can  be  assumed. 

-  If  the  same  [alternative]  is  preferred  for  all... budgets,  that  system  is 
dominant. 

-  If  the  same  [alternative]  is  not  dominant,  the  use  of  several... budgets  is 
nevertheless  an  essential  step,  because  it  provides  vital  information  to 
the  decision  maker.”  (p.176) 

Hitch  C.  &  R.  McKean  1967  “The  Economics  of  Defense  in  the 
Nuclear  Age”  Harvard  University  Press,  Cambridge,  Mass. 


AoA 


[Marine  Corps  Systems  Command  PA&E  Methodology] 

Statement  of  requirements 

-  Mission  Needs  Statement  (“Customer”),  Subject  Matter  Expert  input,  etc. 
Development  of  alternatives 

-  Complete  and  exhaustive;  Think  broadly 

w-  Multi-step  approach;  eliminate  no  alternative  before  its  time 
1  -  “Do  nothing”  is  an  alternative! 

Evaluation  of  effectiveness  of  alternatives  (Modeling  &  Simulation) 

_  Dorfr\rmonr>Q  /l\/IODV  inhoront  rhoror'+orietir'  r\ f  oltnmoti\/n 


Estimation  of  “rough  order  of  magnitude”  life-cvc  > 


Integration  of  results 

Equal  effectiveness 

•  Set  a  threshold  for  a  given  level  of  effectiveness 

•  Buy  enough  systems  to  achieve;  compare  costs 


-  Equal  cost 

•  Set  a  fixed  expenditure  rate 

•  Compare  effectiveness  with  equal-cost  alternatives 


Conclusions  and  recommendations 


DoD  Outsourcing  Examples 

Materiel  management 

-  Distribution 

-  Inventory  control 

-  Disposal 

Base  commercial  activities 

-  Facilities  maintenance 

-  Food  services 

-  Vehicle  maintenance 
Depot  maintenance 
Finance  &  accounting 

-  Purchasing  &  Travel  (credit)  cards 

-  Debt  &  claims  management 

-  Administrative  support 
Training 

-  Simulators 

-  Distance  Learning 
Information  Technology 

-  Computer  equipment,  maintenance  &  repair 

-  Communication  equipment,  maintenance  &  repair 

-  Software  development 

-  Internet  services 


Summary:  Six  Approaches  to  AoA 

Can  Construct  alternatives: 

1 .  Fixed  Budget  Approach  (Construct  alternatives  with  equal 
budget): 

-  Objective:  Maximize  Effectiveness 

2.  Fixed  Effectiveness  Approach  (Construct  alternatives  with  equal 
effectiveness): 

-  Objective:  Minimize  Costs 

3.  Expansion  Path  Approach  (Construct  alternatives  as  Cost- 
Output/Effectiveness  Relations):  Sensitivity  Analysis 

Can  Modify  pre-determined  alternatives:  “Level  the 
Playing  Field” 

4.  Modified  Budget  Approach  (Identify  high/low  cost  alternative  as 
“revealed”  budget  constraint  and  adjust  others  accordingly) 

-  Objective:  Maximize  Effectiveness  (GOTO  1  &  3) 

5.  Modified  Effectiveness  Approach  (Identify  effectiveness  of  an 
alternative  as  “revealed”  objective  and  adjust  others  accordingly) 

-  Objective:  Minimize  Costs  (GOTO  2  &  3) 

Cannot  Construct  or  Modify  alternatives: 

6.  Opportunity  Cost/Benefit  Approach:  (Inter-program  analysis) 


To  generate  MOE: 

Identify:  decision  scenario,  relevant  players 
“decision  makers”  (DM),  and  time  frame 

•  Identify  relevant  players:  DM 

-  Users 

-  Evaluators  MOE 

-  “Political”  Stakeholders 

—  Payers  $$  (Cost  as  an  Independent  Variable — CAIV  ) 

•  “Top-down”  approach  to  assist  DM  to 
describe  components  of  MOE  (utility)  function 

-  Saaty’s  (1977)  Analytic  Hierarchy  Process  (AHP) 

-  Proceed  from  general  criteria  to  measurable  attributes 


Example:  MOE 


DoD  Outsourcing  Goals 

1.  Cut  Costs: 

-  Competition  increases  productivity  and  cuts  costs. 

-  Leverage  economies  of  scale  &  scope,  learning  curves, 
and  specialized  human  capital  and  technology  investments. 

2.  Boost  Performance/Effectiveness: 

-  Continuous  improvement  of  product  and  service  quality, 
schedules,  and  responsiveness  to  military  demands. 

3.  Focus  on  Core  Competencies: 

-  Focus  scarce  DoD  resources  and  defense  management 
attention  on  core  competencies. 

-  Provide  oversight  and  monitoring  of  service  and  supply 
contracts,  and  preserve  option  of  future  competitions. 


Federal  Outsourcing  Guidance 


Office  of  Management  &  Budget  (OMB  Circular  A-76) 


•  Federal  Acquisition  Streamlining  Act  (PL  103-355) 

•  Federal  Acquisition  Reform  Act  (PL  104-106) 

•  Information  Technology  Management  Reform  Act  (PL 
104-106) 


•  Federal  Activities  Inventory  Reform  Act  (PL  105-270) 


Federal  Acquisition  Regulations  (FAR/DFAR  5000. 1&2) 


_ OMB  Circular  A-76 _ 

•  “Mandates... the  government  obtain  commercially 
available  goods  and  services  from  the  private  sector 
when  it  makes  economic  sense  to  do  so.” 

•  “[R]equires... structured  process  for  [evaluating]  the 
most  efficient  and  cost-effective  method  of 
performance  for  commercial  activities...” 

Four  Steps: 

1 .  Develop  Statement  of  Work  (SOW)  or  Performance  Work  Statement  (PWS)  to 
define  desired  performance/effectiveness  (and  a  Quality  Assurance  Surveillance 
Plan— MOE) 

2.  Construct  Most  Efficient  Organization  (MEO)  for  in-house  competitor 

3.  Issue  Invitation  for  Bid  (IFB)  for  well-defined,  routine  commercial  activities;  or 
Request  for  Proposal  (RFP)  for  less  well-defined,  more  complex  activities 

4.  Source  Selection:  Compare  bids  or  proposals — ’’least  cost”  for  IFB; 


Share  A-76,  Process  Overview, 


“BEST  VALUE”  for  RFP 


12/22/03 


Federal  Activities  Competed 
Most  Frequently  in  FY  2004-5 


Activity 

FTE 

% 

Maintenance/property 

mgmt 

5,459 

29 

Logistics 

4,435 

23 

Information  technology 

3,262 

17 

HR/personnel  mgmt 

1,378 

7 

Finance  &  accounting 

1,178 

6 

Administrative  support 

1,078 

6 

Other 

2,316 

12 

OMB  (2006)  “Competitive  Sourcing,”  Executive  Office  of  the  President,  p.10 


Examples:  Outsourcing  Travel 

(2005  Vendor  Sales  to  the  Federal  Government) 


Airlines 

2005  ($mil) 

Market  share 

United 

$846 

25% 

Delta 

$718 

21% 

American 

$491 

14.4% 

Cars 

2005  ($mil) 

Market  share 

Hertz 

$76 

20.2% 

Enterprise 

$56 

15% 

Avis 

$54 

14.3% 

Hotels 

2005  ($mil) 

Market  share 

Marriot 

$146 

7.3% 

Holiday  Inn 

$141 

7.0% 

Residence 

Inn 

$125 

6.3% 

Gov’t  Executive 

8/15/06  pp.66-8 

2004 

2005 

Defense 

$8.9bil 

$10.9bil 

Homeland 

Security 

$849mil 

$940mil 

Outsourcing  Lessons  Learned 

1 .  Use  performance-based  contracting 

-  Do  not  list  tasks  [mix  of  inputs],  but  state  results 
sought  or  problems  to  be  solved  [desired 
attributes/characteristics  of  outputs/outcomes] 

-  Tell  them  WHAT  you  want. . .not  HOW  to  do  it. 

2.  Choose  contractors  according  to  BEST 
VALUE 

-  Source  Selection:  Trade  off  performance  and 
price  instead  of  simply  awarding  to  the  lowest 
bidder. 


J.  Gansler  &  R.  Lipitz  (2003)  “Moving  Toward  Market-Based  Government,”  IBM 
Endowment  for  the  Business  of  Government  (p.  1 5) 


EEoA:  Vendor  Expansion  Paths 

Maximize  Attribute  Bundle  subject  to  Budget  Constraint 

(Assumptions  for  llustration:  Identical,  constant,  technology  (i.e.  Dll  =  D12  =  D1  and 
□21  =  D22  =  D2),  constant  returns  to  scale  (i.e.  CRTS  =>  □  1  +  D2  =  1 ,  or  if  □  1 
=  □  then  D2  =  1-  □),  constant  attribute  costs,  and  cl  1/c21  >  c12/c22  or  cl  1/c12 
>  c21/c22,  so  slope  of  vendorl  expansion  path  is  steeper  than  vendor  2. 


(  /ci2m 


DECISION  SCIENCES  APPROACH:  Exogenous  Alternatives 
Example  of  “SUPERIOR  (Dominant)  SOLUTION” 


Source  Selection  Decision:  Eliminate  A2 ! 


▲ 

MOE(Utils) 


C/3 

xn 


o 


► 

Budget  ($) 


ECONOMIC  APPROACH:  Evaluation  of  Alternatives  (“Engel  Curves”) 
3.  Expansion  Path  (Response  Function)  Approach 
Do  not  eliminate  A2  prematurely:  Explore  impact  of  budget 

cuts  (Identify  vendor  responses) 


▲ 

MOE(Utils) 


GO 

GO 


o 


A2 


Bl 


► 

Budget  ($) 


ECONOMIC  APPROACH:  Endogenous  Alternatives  (“Engel  Curves”) 
3.  Expansion  Path  (Response  Function)  Approach 
(Alternatives  are  Cost-Effectiveness  Relations,  not  Points) 
Explore  impact  of  budget  cuts  (Identify  vendor  responses) 


Source  Selection  Decision:  A2  for  pessimistic  budget;  A1  for  optimistic  budget 


Example:  Evaluate  Alternative  Radar 

Maintenance  Packages 

0.70 
0.60 
0.50 

lu  0-40 
O 

S  0.30 
0.20 
0.10 
0.00 


Rad 

Radar  2 

o 

Radar  1 

o 

Radar  4 
ar  3  q 

$0  $20  $40  $60  $80 

Discounted  LCC 


MOE  =  f(Availability;etc.) 


Which  is  “Best  Value?”Radar  1  or  2? 

(MOE  =  f(Availability;etc.)) 


0.70 


0.60 


0.50 


0.40 


0.30 


0.20 


0.10 


0.00 


Rad 

Radar  2 

Radar  1  T 

Radar  4 
ar  3  q 

o 

Attributes  Radar  1  Radar  2 

Change 

Availability 

0.8 

0.9 

'  ai 

COST  |  $14.00  |  $35.00 

$21.00 

$0  $20  $40  $60  $80 

Discounted  LCC 


Is  it  worth  $21  mil  over 
the  life  of  the  system  to 
obtain  the  extra 
Availability  MOE? 


Source  Selection 


“One  straightforward  method  for  combining  cost  and 
effectiveness  involves  constructing  a  ratio.”  (p.6-3) 

“The  methods  we  choose  for  combining  effectiveness 
[and]  cost... depend  upon  the  nature  of  the  problem. 

We  can  fix  either  cost  or  effectiveness.” 

“If  neither  can  be  fixed... we  can  establish  a 
cost/effectiveness  ratio. . .”  (p.6-10) 

C.  Murray,  Editor  (2002)  Executive  Decision  Making,  National  Security  Decision  Making  Dept., 

U.S.  Naval  War  College  (  )  6th  Edition 

Danger  in  applying 
Bang/Buck  or  Buck/Bang 
Ratios  to  rank  alternatives 


Is  A1  really  superior  to  A2  ? 

LESSON:  DANGER  in  using  Benefit/Cost  (Bang/Buck)  or 
Cost/Benefit  (Buck/Bang)  ratios  without  anchoring  Budget  or  MOE 


MOE(Utils) 


G n 


o 


Benefit/Cost 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

'  / 
/  / 


RANKING: 

(A1)  >  Benefit/Cost  (A2) 

/ 

/ 


✓ 

✓ 

/ 

/ 

/ 


Efficient  Set 


Marginal  Benefit/Marqinal  Cost 

“The  perceived  benefits  of  the 
higher  priced  proposal  shall 
merit  the  additional  cost...” 

FAR  15.101-1  (2)c 


Cost 


